A multimodal interpretation training evaluation method and evaluation device

Abstract

The present application relates to the field of computer technology, and is a speech recognition solution, specifically a multimodal interpretation training evaluation method and evaluation device. The embodiment provided by the present invention determines the semantic correctness and emotional tendency of the translation by performing standard text matching and emotional tendency score calculation on the text data respectively, extracts multiple features from the audio data and determines the translation fluency based on the expression of multiple features, and determines the corresponding semantic score, emotional score and fluency score according to the semantic correctness, emotional tendency and translation fluency, and evaluates the trainee's translation behavior according to the semantic score, emotional score and fluency score. The method provided in the embodiment of the present application can realize the automated evaluation of the translation process and results based on natural language processing technology and speech processing technology, and can comprehensively evaluate the results from three aspects: accurate semantic expression, emotional tendency and fluency expression compared with the existing technology.

Description

Technical Field

[0001] The present application relates to the field of computer technology and is a speech recognition solution, specifically to a multimodal interpretation training evaluation method and evaluation device. Background Art

[0002] Currently, in the field of translation instruction, especially interpreting instruction, error correction and training results scoring for learners primarily rely on one-on-one instruction from teachers, who rely on their experience to perform corrections and scoring. While interpreting ability characteristic scales are currently being developed to achieve objective evaluation, the effectiveness of these scales requires teachers to strictly adhere to their usage rules and evaluation criteria. Therefore, evaluations based on these scales still cannot avoid subjective judgment and are time-consuming. Summary of the Invention

[0003] To address the above technical issues, the present invention provides a multimodal interpretation training evaluation method and device. By obtaining the expression results of the textual features and speech features of the interpreted speech information and combining the expression results of the two features to determine the final evaluation result, compared with the existing technology, it can comprehensively and automatically evaluate the results from three aspects: semantic accuracy, emotional tendency, and fluency expression. To achieve the above objectives, the technical solutions adopted by the present invention are as follows:

[0004] In a first aspect, a multimodal interpretation training evaluation method is provided, the method comprising: generating a corresponding original text to be translated and a corresponding historical corpus according to a training task, delivering the original text to be translated and receiving speech data to be evaluated that matches the original text to be translated; the historical corpus includes historical corpus data corresponding to the training task generated during a historical interpretation training process, and also includes real-time corpus data generated in real time as the training task progresses; extracting text features and speech features from the speech data to form corresponding text data and audio data; extracting the text features comprises: extracting initial text features from the speech data, performing error correction and updating processing on the initial text features based on a historical corpus database to obtain target text features, and forming a target text as the text data based on the target text features; determining a semantic score based on a match between the text data and a standard text, determining a fluency score based on a feature distribution result corresponding to the audio data, and determining a sentiment score based on a sentiment similarity between the text data and the standard text; and determining a scoring result for the speech data to be evaluated based on the semantic score, the fluency score, and the sentiment score.

[0005] In some specific implementations, extracting text features of the speech data to form corresponding text data includes: extracting initial text features of the speech data and forming a corresponding initial text sequence, updating the initial text features according to the similarity distribution between the initial text features and the corpus data in the historical corpus to generate target text features, and generating corresponding text data based on the target text features.

[0006] In some specific implementations, the initial text features are updated based on the similarity distribution between the initial text features and the corpus data in the historical corpus, including: obtaining the similarities between the initial text features and multiple historical corpus data and multiple real-time corpus data respectively, screening the multiple similarities to determine the initial similarity distribution result, and then updating the initial similarity distribution result based on the update weights of the historical corpus data and the real-time corpus data.

[0007] In some specific implementations, the method further includes: obtaining a query vector corresponding to the initial text feature, and a key-value pair corresponding to each semantic feature in multiple historical corpus data and multiple real-time corpus data, and returning at least three closest key-value pairs based on each token in each query vector.

[0008] In some specific implementations, determining the degree of matching between the text data and the standard text includes: obtaining a matching ratio of matching word segments between the text data and the standard text, and determining the matching degree and the corresponding semantic score based on the matching ratio.

[0009] In some specific implementations, obtaining the matching ratio of matching word segments between the text data and the standard text includes: obtaining an alignment vector between the text data and the standard text, and fusing the alignment vector with the word embedding vector of the text data, and pooling the fusion results to determine the word segments in the text data that match the standard text, and counting the ratio of the matching word segments.

[0010] In some specific implementations, obtaining the alignment vector between the text data and the standard text includes: dividing the text data and the standard text into blocks, and determining the context information corresponding to the segmented words within the blocks and the upper and lower block information between blocks, and obtaining the alignment vector between the text data and the standard text based on the context information corresponding to the segmented words and the upper and lower block information corresponding to the blocks.

[0011] In some specific implementations, the speech features include energy features, fundamental frequency change features, and syllable boundary clarity features; extracting the speech features of the speech data includes: sequentially obtaining the energy distribution state, fundamental frequency change distribution state, and syllable boundary clarity distribution state corresponding to multiple frames of data in the speech data.

[0012] In some specific implementations, the fluency score is determined based on the feature distribution results corresponding to the audio data, including: respectively obtaining the sub-scores corresponding to the energy distribution state, the fundamental frequency change distribution state, and the syllable boundary clarity distribution state, and performing weighted summation of the sub-scores based on the corresponding weights to obtain a total score as the fluency score.

[0013] In a second aspect, a multimodal interpretation training and evaluation device is provided, which includes: a data acquisition module for generating a corresponding original text to be translated and a corresponding historical corpus according to a training task, and delivering the original text to be translated and receiving speech data to be evaluated that matches the original text to be translated; a data processing module for extracting text features and speech features of the speech data to form corresponding text data and audio data; the extraction of the text features includes: extracting initial text features of the speech data, and performing error correction and update processing on the initial text features based on a historical corpus database to obtain target text features, and forming a target text as the text data based on the target text features; an evaluation module for determining a semantic score based on the matching degree between the text data and a standard text, determining a fluency score based on a feature distribution result corresponding to the audio data, and determining a sentiment score based on the sentiment similarity between the text data and the standard text; and determining a scoring result of the speech data to be evaluated based on the semantic score, the fluency score, and the sentiment score.

[0014] In the technical solution provided by the embodiment of the present application, the text data and audio data of the trainee's speech data are obtained, and the text data is respectively subjected to standard text matching and sentiment tendency score calculation to determine its translation semantic correctness and sentiment tendency, multiple features are extracted from the audio data and its translation fluency is determined based on the expression of multiple features, and the corresponding semantic score, sentiment score and fluency score are determined according to the translation semantic correctness, sentiment tendency and translation fluency, and the trainee's translation behavior is evaluated according to the semantic score, sentiment score and fluency score. The method provided by the embodiment of the present application can realize the automated evaluation of the translation process and results based on natural language processing technology and speech processing technology, and can comprehensively evaluate the results from three aspects: accurate semantic expression, sentiment tendency and fluency expression compared with the existing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0016] The methods, systems, and / or programs in the accompanying drawings will be further described according to exemplary embodiments. These exemplary embodiments will be described in detail with reference to the drawings. These exemplary embodiments are non-limiting exemplary embodiments, wherein example numerals represent similar structures in the various views of the drawings.

[0017] Figure 1 This is a structural diagram of the teaching system provided in an embodiment of the present application.

[0018] Figure 2 This is a schematic diagram of the server structure provided in an embodiment of the present application.

[0019] Figure 3 This is a flowchart of the training evaluation method provided in the embodiment of the present application.

[0020] Figure 4 It is a schematic diagram of the structure of the evaluation device provided in the embodiment of the present application.

[0021] Figure 5 This is a schematic diagram of the terminal device structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to better understand the above technical solution, the technical solution of the present application is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations on the technical solution of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.

[0023] In the following detailed description, numerous specific details are set forth by way of example in order to provide a thorough understanding of the relevant teachings. However, it will be apparent to one skilled in the art that the present application can be practiced without these details. In other instances, well-known methods, procedures, systems, compositions, and / or circuits have been described at a relatively high level, without detail, to avoid unnecessarily obscuring aspects of the present application.

[0024] Flowcharts are used in this application to illustrate the execution processes performed by the system according to the embodiments of the present application. It should be clearly understood that the execution processes of the flowcharts may not be executed in sequence. Instead, these execution processes may be executed in reverse order or simultaneously. In addition, at least one additional execution process may be added to the flowchart. One or more execution processes may be deleted from the flowchart.

[0025] Before further explaining the embodiments of the present invention in detail, the nouns and terms involved in the embodiments of the present invention are explained. The nouns and terms involved in the embodiments of the present invention are subject to the following interpretations.

[0026] (1) In response to, it is used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.

[0027] (2) Based on, used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.

[0028] The embodiment of the present application provides a multimodal interpretation training evaluation method applied to a teaching system, for automatically evaluating the results of a trainee's interpretation process.

[0029] In this embodiment, the overall structure of the teaching system 10 can be found in Figure 1 As shown, it includes at least a server 11, a student terminal 12 and a teacher terminal 13. Among them, the student terminal and the teacher terminal can be mobile terminals, including but not limited to smart phones, tablet computers and other mobile devices that can carry applications. In this embodiment, smart phones are preferred. In this embodiment, the structure of the server 11 can be referred to. Figure 2, is provided with a data processing unit 111 and a database 112, and content is displayed through display interfaces on the student side and the teacher side respectively. In the interpretation training scenario, the system selects the interpretation project to be trained and the corresponding audio data on the student side. After being retrieved from the database, it is sent to the terminal device and the trainee's voice is obtained and recorded through the sound recording hardware in the student side. The obtained sound data is transmitted to the data processing unit in real time, and the interpretation training evaluation method configured in the data processing unit is used to process the trainee's interpretation evaluation results; and the interpretation evaluation results and the recorded interpretation sound files are stored in the database and the corresponding teacher side, where the interpretation evaluation results include the scoring results, as well as the text display of the interpretation results and the display of important marks. The teacher side is used to manually update the interpretation evaluation results and synchronize them to the student side.

[0030] In this embodiment, the database includes a text database, a historical corpus database, and an emotional corpus database. Among them, the text database stores the original audio data to be translated related to the training task. Different training tasks are configured with corresponding original audio data to be translated, which are retrieved according to the training task. The historical corpus database contains the historical corpus data corresponding to the training task, as well as the real-time corpus data generated in real time as the training task is carried out. The emotional corpus data stores text features related to emotional words and corresponding emotional polarity scores. For this emotional corpus database, a database that has been constructed and open sourced in the prior art can be used, such as the Hownet emotional dictionary, the General Inquirer emotional dictionary, etc.

[0031] In existing technologies, online teaching, especially interpreting, faces significant scenario and technical limitations due to the scarcity of teacher resources. This makes it difficult for teachers to provide timely feedback and ratings on students' progress, leading to high training and teaching costs. With the advancement of computer technology, especially speech recognition, it is now possible to convert interpreters' interpretations into text based on computer speech recognition.

[0032] However, it is worth noting that with regard to the characteristics of translation, especially interpretation, the current technical solutions can only convert the trainee's interpretation results from sound to text, and are unable to judge the interpretation results. In particular, in terms of automatic oral scoring methods, the scoring methods at the speech level are basically mature, such as whether the trainee's pronunciation is accurate. However, there are not many scoring methods at the content level, such as whether the trainee's expression is complete, whether the wording is accurate, whether the sentence is meaningful, and whether the emotional expression of the sentence is accurate. Moreover, for interpretation scenarios, different types of languages, different trainees, their pronunciation habits, and accent problems will cause recognition errors. Therefore, providing an evaluation method can improve the recognition of the trainee's interpretation results in interpretation scenarios, and evaluate and score the interpretation results according to the judgment criteria at the semantic level.

[0033] In order to achieve the above technical effects, the teaching system in this embodiment is equipped with an interpretation training evaluation method in the data processing unit. The real pronunciation result of the acquired trainee's real-time interpretation data is determined based on context association and database association to generate interpretation text data to improve the recognition accuracy. The semantic relevance is determined based on the context semantics and the standard translation text is compared. The fluency is determined based on the sound spectrum characteristics. The trainee's interpretation results and interpretation process are comprehensively scored based on multimodal data fusion.

[0034] Specifically, please refer to 2 for the interpretation training evaluation method in this embodiment, which includes the following steps:

[0035] Step S31: Generate a corresponding original text to be translated and a corresponding historical corpus according to the training task, deliver the original text to be translated, and receive speech data to be evaluated that matches the original text to be translated.

[0036] In this embodiment, the training task is selected and obtained by the trainee on the student side. A plurality of training tasks are configured on the display interface of the student side. The target training task is determined by selection on the student side, and the training task is sent to the data processing unit. The data processing unit retrieves the corresponding original text to be translated and the historical corpus from the database based on the link or label corresponding to the training task.

[0037] Among them, the original text to be translated will be played to the trainee through the sound hardware of the terminal device on the student side and at the trainee's choice. The historical corpus is the historical corpus data generated by the historical interpretation training process related to this training task, and also includes real-time corpus data generated in real time based on the implementation of this training task. The training task is constructed by multiple labels. For example, for a training task, a scene label can be selected, and the original text to be translated corresponding to the above label is generated from the database according to the scene label. Among them, the historical corpus database is the historical process data corresponding to this label, that is, the training task, which is used for subsequent speech recognition text conversion.

[0038] Specifically, automatic speech recognition technology is used to perform real-time text conversion upon acquiring the trainee's speech data. However, in actual use, factors such as the complexity and diversity of speech, the speaker's accent, speech rate variations, and background noise interference make it difficult to accurately recognize words with similar pronunciations, leading to deviations in semantic understanding. For example, the standard pronunciation of "Zhejiang" (zhejiang) may result in "folding general" (zhejiang), while the standard speech "How is the weather in Taizhou, Zhejiang today?" (zhejiangtaizhou) may be distorted into "How is the weather in Taizhou, Zhejiang today?" (zhejiangtaizhou). Furthermore, the English speech data "in this monotonous life of mine that was a pleasant event" (mind) may be mistakenly recognized as "mind" (mind) due to the similar pronunciations of "mind" and "mine," especially for continuous speech. Therefore, to prevent the impact of recognition errors on subsequent evaluation accuracy, in this embodiment, errors in speech-to-text conversion need to be corrected to obtain corrected recognition results.

[0039] However, it is worth noting that the speech recognition error correction process not only needs to consider whether a single word is correct but also needs to consider the text within the entire document. This error correction not only needs to consider the correctness of a single sentence, but also needs to consider other sentences in the entire document, so as to improve the overall accuracy and readability of the text. Specifically, a larger range of context-based connections is needed for error correction, which not only needs to consider the correctness of the correction vocabulary and grammar, but also needs to determine the semantic relationship between sentences in the document and the consistency of the overall document structure. In general, it is usually necessary to obtain the context attention weight corresponding to each word and judge whether the word is wrong and perform error correction based on its attention weight. However, in this embodiment, in order to enrich the semantics of the context, a historical corpus database is configured, and the context attention weight of the currently converted word and the corpus data in the historical corpus database are determined, thereby improving the error correction capability.

[0040] In this embodiment, the historical corpus database is configured before the task is performed, and comprehensive information of the entire document or data set can be accessed when the task is performed. In addition, as the training task is carried out, the process data generated in each training task can be accumulated, and these accumulated data serve as the basis for storage. The historical corpus database in this embodiment enables the data processing unit to access the entire document, as well as collect real-time corpus data and related contextual information from the data, including all sentences before and after the current sentence. This enables a better understanding of the semantic coherence and contextual relationships within the document during the speech recognition process, thereby improving the ability to process and analyze information in the document.

[0041] Step S32. Extract the text features and voice features of the speech data to form corresponding text data and audio data; the extraction of the text features includes: extracting the initial text features of the speech data, and performing error correction and update processing on the initial text features based on a historical corpus database to obtain target text features, and forming a target text as the text data based on the target text features.

[0042] The text features and speech features of the acquired trainee's speech data are extracted to serve as input data for subsequent training evaluation. The text feature extraction process first performs text conversion to form initial text features. During the text conversion process, the initial text features are corrected and updated using the historical corpus database in step S31 to obtain target text features, thereby forming the target text. The resulting target text is the text data.

[0043] Among them, the above text conversion and text update process first converts the speech into text through the existing ASR model. The ASR model can adopt the structure of the existing technology. The ASR model can convert the input speech data into text in real time to obtain the initial real-time text of the speech data, and organize the initial real-time text into an initial real-time text sequence based on the time series. ;in, Represents the i-th word in the sequence.

[0044] The obtained initial real-time text sequence is corrected based on end-to-end error correction and historical corpus database to obtain the target text.

[0045] The processing logic of the error correction process is to determine the similarity distribution between the initial text features corresponding to the initial text and the expected data in the historical corpus, and update the initial text features according to the similarity distribution to generate target text features, and generate corresponding text data based on the target text features. In this embodiment, the determination of the similarity distribution is based on the attention score, that is, by determining the attention score corresponding to the initial text features in the input initial text sequence relative to each corpus data in the historical corpus database, and determining whether the current initial text features are incorrect based on the corresponding attention scores, and using the text with the highest attention score as the target text feature.

[0046] In this embodiment, error correction of the initial text not only considers individual sentences, but also other sentences in the entire document to improve the overall accuracy and readability of the text. Specifically, error correction in this embodiment requires a wider range of contextual awareness and the ability to handle more complex errors. Therefore, the error correction process in this embodiment needs to utilize a wider range of contextual information, including not only the correctness of vocabulary and grammar, but also the semantic relationship between sentences in the document and the consistency of the overall document structure.

[0047] Therefore, the update or error correction process in this embodiment uses as comparison information sources not only the contextual relationship in the current initial text sequence, but also the entire data set including the entire current document, namely the historical corpus data.

[0048] Specifically, by obtaining the first type similarity and the second type similarity between the initial text features and multiple historical corpus data and multiple real-time corpus data in the corpus database, and screening the multiple first type similarities and the second type similarities to determine the initial first type similarity and the initial second type similarity corresponding to the above two types of similarities, and based on the update weights corresponding to the first type similarity and the second type similarity, the initial first type similarity and the initial second type similarity are updated, the result with the highest similarity is used as the target similarity and the corresponding text features are updated and corrected for the initial text features. In this embodiment, the update weights for the historical corpus data and the real-time corpus data are 0.4 and 0.6 respectively. When the historical corpus data and the real-time corpus data have the same similarity to the initial text and the two have different text features, the real-time corpus data is preferred. That is, context information is preferentially selected as the perception guide of semantics.

[0049] Specifically, the above-mentioned processing process is implemented by providing a processing model in this embodiment. This model includes a coding and decoding unit, wherein the encoder is constructed based on the Transformer and a multi-head attention mechanism is configured in the encoder to obtain the first attention matrix corresponding to each word in the current input initial text sequence. It can be understood that the encoder is used to obtain the contextual semantic information in the current sequence corresponding to each word in the input initial text sequence. And the decoder is used to obtain the contextual semantic information in the historical corpus data and the real-time corpus data corresponding to each word in the initial text sequence. Among them, the contextual semantic information is output based on the attention matrix, and the attention matrix is ​​the similarity distribution of the final output.

[0050] It is worth noting that the contextual semantic information corresponding to the real-time corpus data in the historical corpus is different from the contextual semantic information corresponding to the encoder. The contextual semantic information in the historical corpus includes the contextual semantic information in the current text sequence and the contextual semantic information corresponding to the processed text sequence; while the contextual semantic information in the encoder is only the contextual semantic information of the current text sequence.

[0051] The decoder is also constructed using the Transformer structure. Furthermore, because the decoder is used to obtain the attention matrix corresponding to the current initial text and historical and real-time corpus data, and the amount of historical corpus data is relatively large, in order to quickly process text features, this embodiment configures a k-value proximity search mechanism during calculations in the decoder to reduce search costs.

[0052] Specifically, the encoder first processes the initial text sequence based on the word segmenter and converts it into a series of tokens, where the tokens may be words, subwords or characters. Then, each token is mapped to a unique ID, which forms a sequence. The token ID is converted into the corresponding embedding vector, which contains word embedding and position encoding to provide the model with information about the order of words. The embedded sequence passes through the encoder layer to generate query vectors, key vectors and value vectors. Based on multi-head attention, the query vectors, key vectors and value vectors are mapped to different dimensional spaces through multiple different linear mappings to form different query vector groups, key vector groups and value vector groups. The similarity between elements is obtained by the dot product of the query vector and the key vector in the query vector group and the key vector group, and the output weight is obtained by normalization through the softmax function. Finally, the weighted sum of all value vectors of the output of each text is obtained to obtain the output vector, which is the first output attention matrix.

[0053] In addition, the decoder is also configured with a Transformer module, which is used to receive each token ID in the query vector obtained from the encoder, and through similarity matching, returns the top k closest key pair values ​​according to each token ID to determine the corpus data closest to the historical corpus.

[0054] In this embodiment, the key-value pairs are screened based on the k1 key-value pairs returned by a k-nearest neighbor search operation against the historical corpus data for each token ID, and the k2 key-value pairs returned by a k-nearest neighbor search operation against the real-time corpus data for each token ID. The similarities of the k1 and k2 key-value pairs are then updated using the first and second type weights, respectively, to obtain the corresponding similarities for the key-value pairs, and the top k key-value pairs are screened. Here, k1, k2, and k are all 3.

[0055] Through the above filtering, we can obtain each query vector and its corresponding three key-value pairs. We then calculate the dot product between each query vector and its retrieved key to obtain the attention matrix. We then calculate the weighted sum of the attention matrix and the retrieved value to form the second attention matrix for the decoder.

[0056] The first attention matrix and the second attention matrix obtained through the above processing are weightedly merged through the gated hyperparameter g to balance the local context information and external context information to obtain the target attention matrix, and the input speech data is sorted and organized based on the target attention matrix to obtain the text data corresponding to the final target text features.

[0057] The embodiment of the present application combines contextual semantics with the semantics corresponding to the constructed corpus to correct recognition errors in the current converted text to obtain text data that conforms to the contextual semantics, thereby improving the accuracy of speech-to-text conversion and thus improving the accuracy of subsequent evaluations.

[0058] As can be seen from the above, text features in this embodiment refer to representations of text, while speech features in this embodiment refer to representations of speech. While speech can have various types and forms of feature representation, speech features in this embodiment are used to indicate changes in the current speech and are used for subsequent fluency assessment.

[0059] Therefore, in order for the subsequent speech features to intuitively reflect the fluency of the voice, in this embodiment, energy features, fundamental frequency variation features, and syllable boundary clarity features are used as speech features.

[0060] Specifically, for the extraction of the above three features, it is first necessary to perform window framing processing on the speech data to obtain multiple frame signals.

[0061] Among them, the energy characteristics are obtained by calculating the sum of the squares of the amplitudes corresponding to each frame signal, and combining the calculation results of all frame signals to determine the energy distribution state corresponding to the voice data. This state is used to evaluate the dwell time and dwell frequency in the voice data.

[0062] To obtain the fundamental frequency variation characteristics, a fast Fourier transform (FFT) is performed on the framed signal, more efficiently separating the discrete one-dimensional speech signal data according to the frequency bands it falls into. This results in a spectral representation of the speech signal. The left and right windows after the FFT are then projected to the vanishing point, and a scaling factor is calculated. This scaling factor represents the rate of change of the current frame. The vanishing point inner product is the process of projecting the data at -T0 and T0 using a finite point τ, and then summing the dot products of the scaled points. The data within the two windows is scaled proportionally and then summed to obtain the compressed windows FL and FR, or the compressed FR and FL. The scaling factors corresponding to all frames are obtained by performing this operation based on the data of all frames. The proportional relationship between the scaling factors represents the fundamental frequency variation characteristics between frames.

[0063] To obtain the syllable boundary clarity distribution, the framed signal is similarly subjected to a fast Fourier transform. The logarithmic energy spectrum is then calculated using a Mel filter bank. The 12-13 dimensional MFCC features are then extracted using a discrete cosine transform, and dynamic changes are captured using first-order and second-order differences. The local mean and local average are calculated for the MFCC sequence sliding window. When the local mean exceeds a preset threshold, the initial boundary is determined. The difference between the local mean and the variance corresponding to adjacent initial boundaries is compared. The difference between adjacent initial boundaries in all frames of the speech data is obtained, and the distribution of differences less than the preset difference threshold is counted. This distribution is the syllable boundary clarity distribution.

[0064] The embodiment of the present application obtains three feature states corresponding to the voice data through the above processing process, which are used to indicate the continuity, intonation variability and clarity of the voice data, and provide a relatively complete feature expression for subsequent evaluation.

[0065] Step S33. Determine a semantic score based on the degree of match between the text data and the standard text, determine a fluency score based on the feature distribution result corresponding to the audio data, and determine a sentiment score based on the sentiment similarity between the text data and the standard text; and determine a scoring result of the speech data to be evaluated based on the semantic score, the fluency score and the sentiment score.

[0066] In this embodiment, the evaluation results are determined based on three dimensions: text accuracy, expression fluency, and sentiment similarity. For text accuracy, the semantic score is determined by comparing the degree of match between the acquired text data and the standard text. For expression fluency, the distribution of the multiple speech features acquired is determined. For sentiment, the similarity between the acquired text data and the standard text is determined.

[0067] The matching degree and sentiment score are determined based on the text data. The matching degree refers to the matching degree between the text data and the standard text, and the sentiment score refers to the difference between the sentiment score corresponding to the text data and the standard text.

[0068] Specifically, in this embodiment, the matching calculation and sentiment score determination are first performed on the text data and the standard text. Because the translation logic in the interpretation process is based on the same sentence, the relationship between the sentences in the final text presentation is clear and one-to-one. Therefore, in this embodiment, the text matching calculation is based on the matching calculation between sentences, and the sentiment score corresponding to the text data and the standard text is also determined based on the sentence.

[0069] Specifically, corresponding sentence pairs in the text data and the standard text are constructed, and the sentence pairs are divided into multiple blocks of the same length, and the corresponding matching degree and sentiment score are determined based on the blocks.

[0070] To determine a match, we first apply intra-block attention to each block to capture local dependencies within the block and obtain the block's meaning. We then process the outputs of all blocks using inter-block attention to obtain the global dependencies of the sentence. Finally, we use cross-attention to obtain block-level interaction information for the sentence pair. This block-level interaction information represents the probability distribution of similarity between each block in the two sentences. This similarity distribution is then statistically analyzed to determine the similarity of the corresponding sentences.

[0071] Among them, the similarity probability distribution refers to the distribution of multiple similarity probabilities corresponding to the current block of text data and all blocks of standard text. The calculation of sentence similarity refers to obtaining the maximum similarity probability corresponding to each block of text data, and counting the proportion of the number of blocks greater than the preset similarity probability threshold in the sentence. For example, when a sentence of text data has five blocks, the maximum similarity probabilities of the first block, the second block, the third block, the fourth block and the fifth block are 75%, 60%, 99%, 85% and 95% respectively. Among them, the similarity probability threshold is 80%, then the number of blocks currently greater than the preset similarity probability threshold is 3 and the proportion of the sentence is 60%, then the similarity of the current sentence is 60%.

[0072] For text similarity, the similarities of the above multiple sentences are averaged to obtain the final text similarity. It is also worth noting that the corresponding blocks with similarity below the similarity probability threshold are highlighted in the corresponding text data and displayed on both the student and teacher sides.

[0073] Among them, the inter-block attention in this embodiment is multi-dimensional additive attention, and its feature extraction and processing can be implemented based on the attention mechanism. The difference is that for a block , each current word in the block is represented as q. For the current word, the correlation between the current word vector and the vector in each block is first calculated to form a score vector. The dimension of the score vector is the same as the word embedding vector. Then, a softmax calculation is performed to obtain the probability distribution. The calculation process of multi-dimensional additive attention is based on the following formula: , where W, 、 is the parameter matrix, and b is the offset, is the score vector of the current query vector and the context vector within the block.

[0074] Specifically, in order to obtain the vector representation of the block, the intra-block global attention is used for the intra-block attention output within each block. The calculation process is based on the following formula: ;in, is the output vector of the intra-block attention for the corresponding word in the block.

[0075] To obtain a global dependency representation for each block, we apply inter-block attention to each block. The computational process is identical to that for intra-block attention, except that the input to inter-block attention is the output of each block for the German-Austrian sentence pair. Finally, a gating mechanism is used to combine the outputs of intra-block global attention and inter-block attention to obtain a block-based representation of the local and global dependencies of the sentence pair, which ultimately represents the similarity distribution of the block of text data relative to the block of standard text.

[0076] The above method can be used to obtain the text similarity of the text and obtain the corresponding semantic score based on the text similarity. The specific method is determined by the mapping score of the interval where the text similarity is located. The mapping score of the interval is a pre-set scoring rule and will not be repeated in this embodiment.

[0077] For sentiment scoring, the target text features obtained in step S32 are first combined based on the sentences to obtain a text feature sequence corresponding to each sentence in the text data. Similarly, for the standard text, the feature extraction method in step S32 is used to determine the text feature sequence corresponding to each sentence in the standard text. The two text feature sequences are then matched against the sentiment corpus database to obtain the sentiment analysis score corresponding to each text feature sequence.

[0078] Specifically, a sentiment corpus database can be understood as a sentiment lexicon. By matching text features with similarity, we determine the sentiment words in each text feature sequence. Based on the sentiment score corresponding to each sentiment word in the sentiment lexicon, we determine the sentiment polarity score in the corresponding text feature sequence and the sum of the sentiment polarity scores for each sentence. Finally, we sum the sentiment polarity scores for each sentence to obtain the total sentiment score for the entire text.

[0079] Through the above, the first sentiment total score and the second sentiment total score corresponding to the text data and the standard text can be determined respectively. However, it is worth noting that the above two total scores are used to represent the sentiment tendencies of the two texts, while the sentiment score known in this embodiment represents the similarity between the text data and the standard text. Therefore, in this embodiment, it is also necessary to map the first sentiment total score and the second sentiment total score to the sentiment score to achieve the difference in the sentiment tendencies between the two texts.

[0080] Specifically, in this embodiment, a difference-score gradient mapping relationship is provided. By obtaining the difference corresponding to the first emotion total score and the second emotion total score, and determining the score corresponding to the current difference based on the difference-score gradient mapping relationship, the score referred to in this embodiment is the emotion score.

[0081] As for the fluency score, the sub-scores corresponding to the energy distribution state, fundamental frequency change distribution state and syllable boundary clarity distribution state determined in step S32 are obtained respectively, and the sub-scores are weighted and summed based on the corresponding weights to obtain the total score, which is the fluency score.

[0082] Specifically, the energy distribution state is used to evaluate the dwell time and dwell frequency in the voice data, so the calculation of the sub-score corresponding to the energy distribution state is determined by obtaining the time length when the energy distribution state in the voice data is less than the state threshold, that is, the energy distribution in the voice data is determined and compared with the state threshold, and the accumulated time of the energy distribution corresponding to the energy distribution less than the state threshold is determined to be the time length, and in this embodiment, a mapping relationship between the time length and the score is configured, and the corresponding sub-score is determined based on the current time length.

[0083] The fundamental frequency variation distribution is used to characterize the trainee's voice intonation. While the pitch of an interpreter is generally stable, the fundamental frequency variation distribution can indirectly indicate any abnormal intonation variations caused by uncertainty and disfluency during the interpreter's interpretation. This sub-score is determined by determining the proportion of the total audio data that contains fundamental frequency variation exceeding a preset threshold. The corresponding sub-score is then determined based on a mapping between this proportion and a preset score.

[0084] The syllable boundary clarity distribution is used to characterize the trainee's interpreting performance. Generally, clarity is the standard for interpretation, and the syllable boundary clarity distribution can identify any expression issues, such as swallowing and unclear speech, in the trainee's audio data. Specifically, this sub-score is determined by determining the proportion of distribution differences that fall below a preset difference threshold. The corresponding sub-score is then mapped between this proportion and a preset score.

[0085] The obtained sub-scores are updated with their corresponding weights and summed to obtain the final fluency score. The weights corresponding to the sub-scores are 0.4, 0.3, and 0.3, respectively.

[0086] In this embodiment, the above method can be used to determine the semantic score and fluency score corresponding to the trainee's interpretation results, and these two scores are used as the final scoring result. In another embodiment, the two scores can be added together to obtain the final score. The calculation method of the score is not further described in this embodiment, and can be set accordingly based on the evaluation habits of the corresponding teacher.

[0087] It is worth noting that during the above processing, unusual words and audio features in the text and audio data are highlighted on the display terminals of the student and teacher terminals. These unusual words refer to words whose similarity is lower than a preset similarity threshold during text matching, and audio data corresponding to features that are lower or higher than a preset threshold.

[0088] The embodiment of the present application provides a multimodal interpretation training evaluation method, which obtains text data and audio data of the trainee's speech data, and performs standard text matching and sentiment tendency score calculation on the text data to determine its translation semantic correctness and sentiment tendency, extracts multiple features from the audio data and determines its translation fluency based on the expression of multiple features, and determines the corresponding semantic score, sentiment score and fluency score based on the translation semantic correctness, sentiment tendency and translation fluency, and evaluates the trainee's translation behavior based on the semantic score, sentiment score and fluency score. The method provided in the embodiment of the present application can realize automated evaluation of the translation process and results based on natural language processing technology and speech processing technology. Compared with the existing technology, it can comprehensively evaluate the results from three aspects: semantic accuracy, sentiment tendency and fluency expression.

[0089] In another embodiment, see Figure 4 In this embodiment, a virtual device, namely, an evaluation device 40, is further configured to execute the process from step S31 to step S33. The evaluation device 40 includes:

[0090] The data acquisition module 41 is used to generate a corresponding original text to be translated and a corresponding historical corpus according to the training task, and deliver the original text to be translated and receive speech data to be evaluated that matches the original text to be translated;

[0091] A data processing module 42 is used to extract text features and voice features of the voice data to generate corresponding text data and audio data;

[0092] The evaluation module 43 is used to determine a semantic score based on the degree of match between the text data and the standard text, determine a fluency score based on the distribution result corresponding to the audio data, determine a sentiment score based on the sentiment similarity between the text data and the standard text, and determine a scoring result of the speech data to be evaluated based on the semantic score, the fluency score and the sentiment score.

[0093] See Figure 5The above method can also be integrated into the provided terminal device 500. In view of the fact that the device may have relatively large differences due to different configurations or performance, it can include one or more processors 501 and memory 502. The memory 502 can store one or more applications or data. Among them, the memory 502 can be a temporary storage or a persistent storage. The application stored in the memory 502 can include one or more modules (not shown in the figure), each of which can include a series of computer-executable instructions in the terminal device. Furthermore, the processor 501 can be configured to communicate with the memory 502, and the terminal device can execute the series of computer-executable instructions in the memory 502. The terminal device can also include one or more power supplies 503, one or more wired or wireless network interfaces 504, one or more input / output interfaces 505, one or more keyboards 506, etc.

[0094] In a specific embodiment, the terminal device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions in the terminal device, and the one or more programs are configured to be executed by one or more processors, including computer-executable instructions for performing the following:

[0095] Generate a corresponding original text to be translated and a corresponding historical corpus according to the training task, deliver the original text to be translated, and receive speech data to be evaluated that matches the original text to be translated;

[0096] Extracting text features and voice features of the voice data to generate corresponding text data and audio data;

[0097] A semantic score is determined based on the degree of match between the text data and the standard text, a fluency score is determined based on the distribution result corresponding to the audio data, a sentiment score is determined based on the sentiment similarity between the text data and the standard text, and a scoring result of the speech data to be evaluated is determined based on the semantic score, the fluency score and the sentiment score.

[0098] Optionally, the processor can execute various functions by running or executing the software program stored in the memory and calling the data stored in the memory, such as executing the above Figure 3 The method shown.

[0099] In a specific implementation, as an embodiment, the processor may include one or more microprocessors.

[0100] The memory is used to store the software program for executing the solution of the present application, and the execution is controlled by the processor. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0101] In this application, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0102] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0103] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0104] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0105] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0106] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A multimodal interpretation training evaluation method, characterized in that: The method comprises: Generate a corresponding original text to be translated and a corresponding historical corpus according to the training task, deliver the original text to be translated, and receive speech data to be evaluated that matches the original text to be translated; the historical corpus includes historical corpus data corresponding to the training task generated during the historical interpretation training process, as well as real-time corpus data generated in real time as the training task progresses; Extracting text features and voice features of the speech data to form corresponding text data and audio data; the extraction of the text features includes: extracting initial text features of the speech data, and performing error correction and update processing on the initial text features based on a historical corpus database to obtain target text features, and forming a target text as the text data based on the target text features; specifically including: obtaining a first type of similarity and a second type of similarity between the initial text features and a plurality of historical corpus data and a plurality of real-time corpus data in a corpus database, and screening the plurality of first type similarities and second type similarities to determine an initial first type similarity and an initial second type similarity corresponding to the above two types of similarities, and updating the initial first type similarity and the initial second type similarity based on update weights corresponding to the first type similarity and the second type similarity, taking the result with the highest similarity as the target similarity and updating and correcting the initial text features with the corresponding text features; A semantic score is determined based on the degree of match between the text data and the standard text, a fluency score is determined based on the feature distribution result corresponding to the audio data, and an emotional score is determined based on the emotional similarity between the text data and the standard text; and a scoring result of the speech data to be evaluated is determined based on the semantic score, the fluency score and the emotional score.

2. The multimodal interpretation training evaluation method according to claim 1, characterized in that: The method also includes: obtaining a query vector corresponding to the initial text feature, and a key-value pair corresponding to each semantic feature in the plurality of historical corpus data and the plurality of real-time corpus data, and returning at least three closest key-value pairs based on each token in each query vector.

3. The multimodal interpretation training evaluation method according to claim 2, characterized in that: Determining the degree of matching between the text data and the standard text includes: obtaining a matching ratio of matching word segments between the text data and the standard text, and determining the degree of matching and the corresponding semantic score based on the matching ratio.

4. The multimodal interpretation training evaluation method according to claim 3, characterized in that: Obtaining the matching ratio of the matching word segments between the text data and the standard text, including: obtaining the alignment vector between the text data and the standard text, and fusing the alignment vector with the word embedding vector of the text data, and pooling the fusion result to determine the word segments in the text data that match the standard text, and counting the ratio of the matching word segments.

5. The multimodal interpretation training evaluation method according to claim 4, characterized in that: Obtaining an alignment vector between the text data and the standard text, including: dividing the text data and the standard text into blocks, and determining context information corresponding to the segmented words within the blocks and upper and lower block information between blocks, and obtaining an alignment vector between the text data and the standard text based on the context information corresponding to the segmented words and the upper and lower block information corresponding to the blocks.

6. The multimodal interpretation training evaluation method according to claim 1, characterized in that: The speech features include energy features, fundamental frequency variation features and syllable boundary clarity features; Extracting the speech features of the speech data includes: sequentially acquiring the energy distribution state, fundamental frequency change distribution state, and syllable boundary clarity distribution state corresponding to multiple frames of data in the speech data.

7. The multimodal interpretation training evaluation method according to claim 6, characterized in that: The fluency score is determined based on the feature distribution results corresponding to the audio data, including: respectively obtaining the sub-scores corresponding to the energy distribution state, the fundamental frequency change distribution state, and the syllable boundary clarity distribution state, and performing weighted summation of the sub-scores based on the corresponding weights to obtain a total score as the fluency score.

8. A multimodal interpretation training and evaluation device, characterized in that: The device comprises: A data acquisition module is used to generate a corresponding original text to be translated and a corresponding historical corpus according to the training task, and deliver the original text to be translated and receive speech data to be evaluated that matches the original text to be translated; A data processing module is used to extract text features and voice features of speech data to form corresponding text data and audio data; the extraction of the text features includes: extracting initial text features of the speech data, and performing error correction and update processing on the initial text features based on a historical corpus database to obtain target text features, and forming a target text as the text data based on the target text features; specifically includes: obtaining a first type of similarity and a second type of similarity between the initial text features and a plurality of historical corpus data and a plurality of real-time corpus data in a corpus database, and screening the plurality of first type similarities and second type similarities to determine the initial first type similarity and initial second type similarity corresponding to the above two types of similarities, and updating the initial first type similarity and initial second type similarity based on the update weights corresponding to the first type similarity and the second type similarity, taking the result with the highest similarity as the target similarity and updating and correcting the initial text features with the corresponding text features; An evaluation module is used to determine a semantic score based on the degree of match between the text data and the standard text, determine a fluency score based on the feature distribution result corresponding to the audio data, and determine a sentiment score based on the sentiment similarity between the text data and the standard text; and determine a scoring result of the speech data to be evaluated based on the semantic score, the fluency score and the sentiment score.

Images