A simultaneous interpretation system evaluation method based on cross-modal semantic similarity

Abstract

The present specification provides a simultaneous interpretation system evaluation method based on cross-modal semantic similarity, which relates to the field of artificial intelligence technology. The method can first collect source speech streams in real time, perform frame processing and system clock synchronization, and obtain first speech. Real-time acquisition of each data block of the to-be-tested simultaneous interpretation system of different output modalities. Merge the data blocks corresponding to the time intervals of each speech segment divided based on the first speech, and summarize the speech segments meeting the alignment quality requirements and the merged time sequence alignment data blocks as evaluation units. Calculate the semantic similarity and semantic similarity score. Without "reference translation", the semantics between the speech segments and the data blocks can be compared, reducing the evaluation cost. Calculate the score of the evaluation unit on each evaluation index in the simultaneous interpretation system evaluation matrix. Based on the weight and score of each evaluation index, determine the quality score of the evaluation unit and the comprehensive score of the to-be-tested simultaneous interpretation system, and improve the reliability of the evaluation result.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to an evaluation method for simultaneous interpretation systems based on cross-modal semantic similarity. Background Technology

[0002] Simultaneous interpretation systems are increasingly used in real-time conferences, international exchanges, and other scenarios. Evaluating the performance and quality of simultaneous interpretation systems is crucial for system optimization, selection comparison, and actual deployment.

[0003] Currently, traditional evaluation methods (such as BLEU and COMET) require human translation or machine-generated "reference translations" as standard answers, and evaluate simultaneous interpreting systems using a single evaluation dimension based on these standard answers. However, human-generated reference translations in simultaneous interpreting scenarios are severely outdated and extremely costly, and machine-generated reference translations may contain omissions, leading to unreliable evaluation results. Furthermore, the single evaluation dimension of traditional methods results in low reliability. Traditional evaluation methods only support the evaluation of text-to-text (S2T) simultaneous interpreting systems, making it difficult to simultaneously evaluate both S2T and S2S systems. For speech-to-speech (S2S) simultaneous interpreting systems, the speech is usually transcribed into text before evaluation, introducing ASR errors and causing evaluation bias. Moreover, traditional evaluation methods require offline evaluation after the entire speech has finished, failing to support real-time evaluation during the speech and lacking real-time capability.

[0004] Based on this, this specification provides an evaluation method for simultaneous interpretation systems based on cross-modal semantic similarity. Summary of the Invention

[0005] To address the problems of existing simultaneous interpretation systems that rely too heavily on reference translations, resulting in high costs, low reliability of evaluation results, limited evaluation dimensions, lack of real-time performance, and difficulty in simultaneously evaluating multiple output modalities (S2T and S2S), this specification provides an evaluation method for simultaneous interpretation systems based on cross-modal semantic similarity to solve the aforementioned problems in the existing technology.

[0006] The following technical solution is adopted in this specification:

[0007] This specification provides a method for evaluating simultaneous interpreting systems based on cross-modal semantic similarity, the method comprising:

[0008] S1: Real-time acquisition of source speech stream, and frame processing and system clock synchronization of the source speech stream to generate audio frames with timestamps, which are combined into the first speech; real-time acquisition of data blocks of the simultaneous interpretation system under test with different output modalities;

[0009] S2: Semantically segment the first speech to obtain each speech segment, merge the data blocks corresponding to the time intervals of each speech segment to generate a time-aligned data block, and summarize the speech segments that meet the alignment quality requirements and the corresponding time-aligned data blocks into an evaluation unit;

[0010] S3: Calculate the semantic similarity between the speech segments and time-aligned data blocks in the evaluation unit, and calculate the semantic similarity score based on the semantic similarity of the evaluation unit;

[0011] S4: Construct a simultaneous interpretation system evaluation matrix. Based on the semantic similarity score, calculate the score of the evaluation unit on each evaluation indicator in the simultaneous interpretation system evaluation matrix. Perform a weighted sum based on the weight and score of each evaluation indicator to obtain the quality score of the evaluation unit.

[0012] S5: Calculate the overall score of the simultaneous interpretation system under test based on the quality score of the evaluation unit.

[0013] Optionally, the output modes of the simultaneous interpretation system under test in S1 include S2T and S2S. When the output mode is S2T, each data block contains text content, the generation timestamp of the text content, and the start and end timestamps of the source speech corresponding to the text content. When the output mode is S2S, each data block contains audio data, the generation timestamp of the audio data, and the start and end timestamps of the source speech corresponding to the audio data.

[0014] Optionally, in step S2, the first speech is semantically segmented to obtain each speech segment, which specifically includes:

[0015] A preset speech activity detection algorithm is used to identify silent segments in the first speech, and silent segments that meet a preset first condition are marked as semantic boundaries. The first condition is that the duration of the silent segment exceeds a preset first threshold.

[0016] Based on the semantic boundaries, the first speech is semantically divided to obtain speech segments.

[0017] Optionally, the method for generating the evaluation unit in S2 is as follows:

[0018] The time interval corresponding to each speech segment is determined as the first interval, and a data block matching each speech segment is found from each data block and designated as the first data block; the time interval corresponding to the first data block overlaps with the first interval.

[0019] According to the chronological order, the first data blocks corresponding to each speech segment are merged to obtain the time-aligned data block corresponding to each speech segment. The time coverage interval corresponding to the time-aligned data block is the union of the time intervals of all data blocks in the time-aligned data block.

[0020] Determine whether the timing alignment data block of each speech segment and each speech segment meet the alignment quality requirements, and summarize the speech segments that meet the alignment quality requirements and their corresponding timing alignment data blocks into an evaluation unit.

[0021] Optionally, determining whether the timing alignment data block of each speech segment and each speech segment meet the alignment quality requirements in step S2 specifically includes:

[0022] The time coverage interval of the time-aligned data block corresponding to each speech segment is taken as the second interval;

[0023] Based on the first and second intervals corresponding to each speech segment, a preset alignment cost function is used to calculate the alignment quality of the temporal alignment data blocks corresponding to each speech segment.

[0024] Determine whether the alignment quality of the temporal alignment data block corresponding to each speech segment and each speech segment meets the alignment quality requirements, wherein the alignment quality requirement is that the alignment quality is not less than a preset second threshold.

[0025] Optionally, the semantic similarity in S3 is calculated based on each cross-modal semantic similarity model, which is used to perform a first type and a second type of task, where the first type is speech to text and the second type is speech to speech.

[0026] The method for calculating semantic similarity in S3 is as follows:

[0027] The data mode of the time-aligned data block in the evaluation unit is determined as the first mode, and the data mode of the speech segment in the evaluation unit is determined as the second mode;

[0028] The first embedding vector of the temporally aligned data block in the evaluation unit is determined by the encoder corresponding to the first modality in each cross-modal semantic similarity model, and the second embedding vector of the speech segment in the evaluation unit is determined by the encoder corresponding to the second modality in each cross-modal semantic similarity model.

[0029] Calculate the semantic similarity between each first embedding vector and each second embedding vector.

[0030] Optionally, the semantic similarity score in step S3 is calculated as follows:

[0031] Each semantic similarity of the evaluation unit is mapped to a specified interval to obtain an initial score;

[0032] The median of each initial score is used as the semantic similarity score.

[0033] Optionally, the method further includes:

[0034] When the output modality of the simultaneous interpretation system under test is S2S and the number of cross-modal semantic similarity models used to perform the second type of task in each cross-modal semantic similarity model is less than a preset third threshold, the temporally aligned data blocks in the evaluation unit are transcribed into text by a preset automatic speech recognition system.

[0035] The semantic similarity between the speech segment and the text in the evaluation unit is determined by using the cross-modal semantic similarity model used to perform the first type of task in each of the cross-modal semantic similarity models.

[0036] Optionally, the method for constructing the simultaneous interpretation system evaluation matrix in S4 is as follows:

[0037] Based on the output mode of the simultaneous interpretation system under test, the evaluation index corresponding to the simultaneous interpretation system under test is extracted from the preset evaluation index set as the target evaluation index.

[0038] Based on the aforementioned target evaluation indicators, an evaluation matrix for the simultaneous interpretation system is constructed. When the output mode of the simultaneous interpretation system under test is S2T, the evaluation matrix includes semantic fidelity, end-to-end delay, terminology standardization, output stability, and fluency. When the output mode of the simultaneous interpretation system under test is S2S, the evaluation matrix includes the semantic fidelity, the end-to-end delay, and the terminology standardization.

[0039] Optionally, calculating the score of the evaluation unit on each evaluation indicator in the simultaneous interpretation system evaluation matrix in step S4 specifically includes:

[0040] When the evaluation metric is semantic fidelity, the final semantic similarity score is used as the score corresponding to semantic fidelity.

[0041] When the evaluation metric is the end-to-end delay, a preset first exponential decay model is used to map the delay time of the evaluation unit into a score, which is used as the score corresponding to the end-to-end delay; the delay time is obtained based on the start time of the speech segment and time-aligned data block included in the evaluation unit;

[0042] When the evaluation metric is terminology standardization, terms are extracted from the time-aligned data blocks included in the evaluation unit, and the matching rate between the extracted terms and the terms in the pre-built domain terminology library is calculated as the score corresponding to the terminology standardization.

[0043] When the evaluation metric is output stability and the number of evaluation units is at least two, the edit distance change rate between the temporally aligned data blocks included in two adjacent evaluation units is calculated, and the edit distance change rate is mapped to a score through a preset second exponential decay model, which is used as the score corresponding to the output stability; the edit distance change rate is obtained based on the length and edit distance of the temporally aligned data blocks in the two adjacent evaluation units;

[0044] When the evaluation metric is fluency, a pre-trained language model for the target language is used to calculate the translation perplexity of the evaluation unit, and the translation perplexity is linearly mapped to a specified interval to obtain the score corresponding to the fluency.

[0045] Optionally, the method for determining the weights of each evaluation indicator in step S4 is as follows:

[0046] When the number of evaluation units reaches a preset threshold, the entropy value corresponding to each evaluation indicator is calculated based on the score of each evaluation unit on each evaluation indicator.

[0047] The weight of each evaluation indicator is determined based on the entropy value corresponding to each evaluation indicator, wherein the entropy value is negatively correlated with the weight.

[0048] Optionally, S5 specifically includes:

[0049] The credibility of the evaluation unit is determined based on the standard deviation of each initial score, and the credibility is negatively correlated with the standard deviation.

[0050] Based on the preset exponential decay coefficient and the standard deviation, the comprehensive weight is determined, and based on the quality score of the evaluation unit and the comprehensive weight, the comprehensive score of the simultaneous interpretation system under test is calculated.

[0051] Optionally, the method further includes:

[0052] Based on the semantic similarity, latency, score on each evaluation indicator, credibility, and overall score of the simultaneous interpretation system under test, the evaluation results in the evaluation result display interface are updated. The evaluation results include a curve of semantic similarity changing over time, a latency curve, a radar chart of scores on each evaluation indicator, a credibility indicator, and a cumulative overall score. The credibility indicator is used to display the color corresponding to the credibility, and the color of the credibility corresponds to different standard deviations within different value ranges.

[0053] An evaluation report is generated based on the evaluation results.

[0054] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects:

[0055] This specification provides an evaluation method for simultaneous interpretation systems based on cross-modal semantic similarity. This method first acquires the source speech stream in real time, performs frame segmentation and system clock synchronization on the source speech stream, generating audio frames with timestamps, which are then combined into a first speech. Data blocks from the simultaneous interpretation system under test in different output modalities are acquired in real time. Through frame segmentation and system clock synchronization, a unified time reference is established for the first speech and each data block, ensuring the accuracy of subsequent data alignment. Furthermore, this invention acquires the source speech stream and data blocks in real time during the presentation to evaluate the system under test in real time, eliminating the need for offline evaluation after the presentation. Compared to traditional evaluation methods, this invention offers high real-time performance. Subsequently, the first speech is semantically segmented to obtain speech segments. The data blocks corresponding to the time intervals of each speech segment are merged to generate time-aligned data blocks. Speech segments that meet the alignment quality requirements and their corresponding time-aligned data blocks are then summarized into evaluation units. This method calculates the semantic similarity between speech segments and time-aligned data blocks in the evaluation unit, and calculates a semantic similarity score based on the semantic similarity of the evaluation unit. It compares the semantics between speech segments and data blocks without any manually or machine-generated "reference translations," solving the evaluation dilemma of "no standard answer," reducing evaluation costs, and enabling unified evaluation of simultaneous interpretation systems across all output modalities. Then, based on the semantic similarity score, it calculates the score of the evaluation unit on each evaluation indicator in the simultaneous interpretation system evaluation matrix, and performs a weighted sum based on the weights and scores of each evaluation indicator to obtain the quality score of the evaluation unit. Based on the quality scores of the evaluation units, it calculates the comprehensive score of the simultaneous interpretation system under test. By using the evaluation indicators in the simultaneous interpretation system evaluation matrix to perform multi-dimensional evaluation of the simultaneous interpretation system under test, rather than relying on a single evaluation dimension, the reliability of the evaluation results is improved.

[0056] In order to ensure the accuracy of the alignment between speech segments and data blocks, this invention employs an alignment cost function to calculate the alignment quality of the temporally aligned data blocks corresponding to each speech segment. The alignment quality is compared with a second threshold to determine whether the alignment is accurate (whether the alignment quality is high). Only when the alignment quality is not less than the second threshold are the temporally aligned data blocks and speech segments merged.

[0057] In this invention, semantic similarity is calculated using multiple cross-modal semantic similarity models. Specifically, the data modality of the temporally aligned data blocks in the evaluation unit is determined as the first modality, and the data modality of the speech segments in the evaluation unit is determined as the second modality. The first embedding vector of the temporally aligned data blocks in the evaluation unit is determined using the encoder corresponding to the first modality in each cross-modal semantic similarity model, and the second embedding vector of the speech segments in the evaluation unit is determined using the encoder corresponding to the second modality in each cross-modal semantic similarity model. The semantic similarity between each first embedding vector and each second embedding vector is then calculated. Calculating semantic similarity simultaneously using multiple cross-modal semantic similarity models effectively suppresses the bias of calculations by a single model, thereby improving the accuracy of semantic similarity and thus enhancing the reliability and robustness of the evaluation results. Furthermore, the cross-modal semantic similarity models can directly handle cross-modal differences between data blocks and speech segments output by the test system in multiple output modalities (S2T and S2S), thereby achieving unified evaluation of simultaneous interpretation systems in all output modalities.

[0058] In this invention, when calculating semantic similarity scores, the median is naturally robust to outliers. Even if an extremely low score is generated due to a mismatch between the data modality of the input data (speech segment or data block) and the input type of a certain cross-modal semantic similarity model, the median can remain stable. Therefore, each semantic similarity of the evaluation unit can be mapped to a specified interval to obtain an initial score, and the median of each initial score can be used as the semantic similarity score.

[0059] In this invention, the evaluation metrics in the evaluation matrix of the simultaneous interpreting system under test differ in dimension or type for different output modalities. Specifically, when the output modality of the simultaneous interpreting system under test is S2T, the evaluation matrix includes semantic fidelity, end-to-end delay, terminology standardization, output stability, and fluency. When the output modality of the simultaneous interpreting system under test is S2S, the evaluation matrix includes semantic fidelity, end-to-end delay, and terminology standardization. This adapts to the evaluation of simultaneous interpreting systems under test with different output modalities. Furthermore, the simultaneous evaluation of multiple dimensions of evaluation metrics can improve the accuracy of the evaluation results.

[0060] In this invention, the weights of each evaluation indicator do not require manual intervention or preset fixed weights. When the number of evaluation units reaches a preset threshold, the entropy value corresponding to each evaluation indicator is calculated based on the score of each evaluation unit on each indicator. The weight of each evaluation indicator is dynamically allocated based on its entropy value. The entropy value reflects the dispersion of the evaluation indicator's score and is negatively correlated with the weight. That is, the smaller the entropy value, the greater the difference between different evaluation units, the more valuable it is for distinguishing the performance of the system under test, and the higher its weight should be. Conversely, the larger the entropy value, the more consistent the scores of the evaluation indicator, the lower its distinguishability, and the lower its weight should be. By adaptively and dynamically allocating weights based on the entropy value of the evaluation indicators, evaluation indicators with higher distinguishability receive higher weights, enabling the evaluation process to automatically adapt to different presentation scenarios.

[0061] In this invention, when calculating the overall score of the simultaneous interpreting system under test, the reliability of the evaluation unit can be determined based on the standard deviation of each initial score. Reliability is negatively correlated with standard deviation. The standard deviation between initial scores is used as a quantitative indicator of multi-model consistency; a smaller standard deviation indicates greater consistency in the initial scores obtained from multiple models, suggesting higher reliability of the evaluation results based on the initial scores. Based on a preset exponential decay coefficient and standard deviation, a comprehensive weight is determined. The overall score of the simultaneous interpreting system under test is calculated based on the quality score of the evaluation unit and the comprehensive weight. By using the standard deviation to weight the quality scores of the evaluation units, evaluation units with high multi-model consistency contribute more to the overall score, while suppressing the contributions of evaluation units with low consistency, thereby improving the accuracy of the overall score.

[0062] In addition, the evaluation results display interface uses a confidence indicator to display different colors to represent the confidence level corresponding to the standard deviation of different value ranges, providing users with an intuitive reference for the reliability of the evaluation results. Attached Figure Description

[0063] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0064] Figure 1 This is a flowchart illustrating an evaluation method for simultaneous interpretation systems based on cross-modal semantic similarity provided in this specification. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0066] This specification provides an evaluation method for simultaneous interpretation systems based on cross-modal semantic similarity. The technical solutions provided by the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0067] Figure 1 This is a flowchart illustrating an evaluation method for a simultaneous interpretation system based on cross-modal semantic similarity provided in this specification, which specifically includes the following steps:

[0068] S1: Real-time acquisition of source speech stream, and frame processing and system clock synchronization of the source speech stream to generate audio frames with timestamps, which are combined into the first speech; real-time acquisition of data blocks of the simultaneous interpretation system under test with different output modalities.

[0069] In this specification, the evaluation device first acquires the source speech stream in real time, performs frame segmentation processing and system clock synchronization on the source speech stream, generates audio frames with timestamps, and combines them into the first speech. Simultaneously, it acquires data blocks from the simultaneous interpretation system under test in different output modalities in real time. The evaluation device can be a server, a system or one or more modules within a system, or an electronic device such as a desktop computer or laptop. For ease of description, the following explanation focuses on the server as the execution entity, illustrating the simultaneous interpretation system evaluation method based on cross-modal semantic similarity provided in this specification.

[0070] The aforementioned source audio stream can be the real-time audio stream captured from the speaker. The source audio stream is sampled at a frequency of 16kHz and in mono PCM format. When performing frame segmentation on the source audio stream, a sliding window is used with a frame length of 25 milliseconds and a frame shift of 10 milliseconds to obtain each audio frame. The system clock synchronization is achieved via NTP with the clock of the simultaneous interpretation system under test, with millisecond-level precision. Each audio frame carries a globally monotonically increasing timestamp. It should be noted that the sampling frequency, format, and system clock synchronization method described above are merely examples, and this specification does not limit the sampling frequency, format, or system clock synchronization method.

[0071] The simultaneous interpretation system under test can be a simultaneous interpretation system with different output modalities, including S2T (speech-to-text) and S2S (speech-to-speech). The server can obtain data blocks of the simultaneous interpretation system under test in real time through API interface or log stream. When the output modality is S2T, each data block contains text content, the timestamp of the text content's generation, and the start and end timestamps of the source speech corresponding to the text content. When the output modality is S2S, each data block contains audio data (PCM format), the timestamp of the audio data's generation, and the start and end timestamps of the source speech corresponding to the audio data.

[0072] The audio frames mentioned above can be stored in the audio pool, and the data blocks mentioned above can be cached in the form of a queue for subsequent processing.

[0073] S2: Semantically segment the first speech to obtain speech segments, merge the data blocks corresponding to the time intervals of each speech segment to generate a time-aligned data block, and summarize the speech segments that meet the alignment quality requirements and the corresponding time-aligned data blocks into an evaluation unit.

[0074] In this specification, the server can semantically segment the first speech to obtain speech segments, merge the data blocks corresponding to the time intervals of each speech segment to generate time-aligned data blocks, and summarize the speech segments that meet the alignment quality requirements and their corresponding time-aligned data blocks into evaluation units. Since the time boundaries of the data blocks output by the simultaneous interpretation system under test may not coincide with the natural semantic boundaries of the first speech, it is necessary to align them to a unified time window. Specifically, when semantically segmenting the first speech to obtain speech segments, the server can use a preset speech activity detection algorithm to identify silent segments in the first speech, and mark silent segments that meet a preset first condition as semantic boundaries. Based on the semantic boundaries, the first speech is semantically segmented to obtain speech segments. The aforementioned speech activity detection (VAD) algorithm can be any existing algorithm used to identify silent segments in audio. The first condition is that the duration of the silent segment exceeds a preset first threshold, which is pre-set, specifically 300 milliseconds. The aforementioned semantic boundaries can divide the first speech into several natural semantic windows (speech segments), i.e. , It indicates the number of speech segments, with each speech segment corresponding to a semantically complete phrase or short sentence.

[0075] When merging the data blocks corresponding to the time intervals of each speech segment to generate time-aligned data blocks, the server can determine the time interval corresponding to each speech segment as the first interval, and search for the data block that matches each speech segment from the data blocks, which is then used as the first data block. The first data blocks corresponding to each speech segment are merged according to chronological order to obtain the time-aligned data block corresponding to each speech segment. Specifically, when searching for the data block that matches each speech segment from the data blocks, the server can perform the following steps for each... In the output queue of the simultaneous interpretation system under test, all data blocks that overlap with the natural semantic window in time are identified. The time interval corresponding to the first data block (obtained based on the generated timestamp) overlaps with the first interval. The time coverage interval corresponding to the time-aligned data block is the union of the time intervals of all data blocks in the time-aligned data block. Furthermore, since the output modality of the simultaneous interpretation system under test is different, the modality of the time-aligned data block is also different. When the output modality of the simultaneous interpretation system under test is S2T, the server can concatenate the text content in the first data block corresponding to each speech segment to obtain the time-aligned data block, whose modality is text. When the output modality of the simultaneous interpretation system under test is S2S, the server can concatenate the audio data in the first data block corresponding to each speech segment to obtain the time-aligned data block, whose modality is speech (audio).

[0076] When summarizing the speech segments that meet the alignment quality requirements and their corresponding time-aligned data blocks into an evaluation unit, the server can determine whether the time-aligned data block of each speech segment and each speech segment meet the alignment quality requirements, and summarize the speech segments that meet the alignment quality requirements and their corresponding time-aligned data blocks into an evaluation unit. Specifically, when determining whether the time-aligned data block of each speech segment and each speech segment meet the alignment quality requirements, the server can use the time coverage interval of the time-aligned data block corresponding to each speech segment as the second interval. Based on the first interval and the second interval corresponding to each speech segment, a preset alignment cost function is used to calculate the alignment quality of the time-aligned data block corresponding to each speech segment. It is then determined whether the alignment quality of the time-aligned data block corresponding to each speech segment and each speech segment meets the alignment quality requirements. Specifically, the alignment cost function is used to measure the alignment quality between the speech segment and the time-aligned data block, and can be calculated using the following formula (1):

[0077] (1)

[0078] in, This indicates the first interval of a speech segment. This represents the time coverage interval of the time-aligned data block corresponding to the speech segment, i.e., the second interval. |.| represents the interval length. The value range is [0,1]. Indicates perfect alignment. This indicates that there is absolutely no overlap.

[0079] The alignment quality requirement mentioned above is that the alignment quality is not less than a preset second threshold, which is preset and can be specifically set to 0.3. When If the alignment is successful and the alignment quality of the speech segment and its corresponding timing alignment data meets the alignment quality requirements, the server can merge the speech segment and its corresponding timing alignment data into one evaluation unit. ,in For speech segments, The data block (text or speech) output by the simultaneous interpretation system under test within the time interval corresponding to the speech segment is time-aligned data. Simultaneously, the server can also record the delay time of the evaluation unit, which is the time difference between the start time of the speech segment in the evaluation unit and the start time of the time-aligned data block in the evaluation unit.

[0080] S3: Calculate the semantic similarity between the speech segments and time-aligned data blocks in the evaluation unit, and calculate the semantic similarity score based on the semantic similarity of the evaluation unit.

[0081] In this specification, the server can calculate the semantic similarity between speech segments and temporally aligned data blocks in the evaluation unit, and calculate a semantic similarity score based on the semantic similarity of the evaluation unit. Specifically, when the speech segments and temporally aligned data blocks in the evaluation unit have the same modality, the server can use a feature extraction model to extract features corresponding to the speech segments and temporally aligned data blocks in the evaluation unit, respectively, and calculate the semantic similarity based on the extracted features. This feature extraction model can be any existing text feature extraction model or speech feature extraction model. However, when the speech segments and temporally aligned data blocks in the evaluation unit have different modalities, the server can use the encoder corresponding to the modality of the speech segment in the cross-modal semantic similarity model to determine the first embedding vector corresponding to the speech segment in the evaluation unit, and use the encoder corresponding to the modality of the temporally aligned data block in the cross-modal semantic similarity model to determine the second embedding vector corresponding to the temporally aligned data block in the evaluation unit, and then calculate the semantic similarity between the first embedding vector and the second embedding vector. The aforementioned cross-modal semantic similarity model can be used to perform the first type of task, which is speech-to-text. The cross-modal semantic similarity model used for performing this task may include an encoder for extracting text features (i.e., a text encoder) and an encoder for extracting speech features (i.e., a speech encoder). This cross-modal semantic similarity model can be any existing model, specifically the BLASER 2.0 model or the SONAR cross-model, or any other cross-modal semantic similarity model; this specification does not impose specific limitations. The semantic similarity can be cosine similarity. Traditional semantic comparison cannot directly handle cross-modal differences between speech and text. This specification employs cross-modal joint embedding space technology, mapping signals from different modalities (i.e., speech segments and time-aligned data blocks) to the same high-dimensional vector space through the cross-modal semantic similarity model. This makes semantically similar speech and text closer in space, and semantically different speech farther apart.

[0082] In addition, the cross-modal similarity model described above can also perform a second type of task, namely speech-to-speech. Therefore, when the speech segments and time-aligned data blocks in the evaluation unit have the same modality, the server can also use the cross-modal similarity model to extract the features corresponding to the speech segments and time-aligned data blocks in the evaluation unit respectively.

[0083] In the above-mentioned calculation of semantic similarity scores based on the semantic similarity of the evaluation units, the server can map the semantic similarity to a specified interval to obtain an initial score, which is then used as the semantic similarity score. The specified interval is pre-set and can be specifically set as follows: The above mapping can be a linear mapping.

[0084] S4: Construct a simultaneous interpretation system evaluation matrix. Based on the semantic similarity score, calculate the score of the evaluation unit on each evaluation indicator in the simultaneous interpretation system evaluation matrix. Perform a weighted sum based on the weight and score of each evaluation indicator to obtain the quality score of the evaluation unit.

[0085] In this specification, the server can construct a simultaneous interpretation system evaluation matrix. Based on semantic similarity scores, it calculates the score of each evaluation unit on each evaluation indicator in the simultaneous interpretation system evaluation matrix. A weighted sum is then performed based on the weights and scores of each evaluation indicator to obtain the quality score of the evaluation unit. Specifically, when constructing the simultaneous interpretation system evaluation matrix, the server can extract the corresponding evaluation indicators for the simultaneous interpretation system under test from a preset set of evaluation indicators, based on the output modality of the system under test. The simultaneous interpretation system evaluation matrix is ​​then constructed based on the target evaluation indicators. The set of evaluation indicators includes multiple indicators. The target evaluation indicators differ for different output modalities of the simultaneous interpretation system under test. Specifically, when the output modality of the simultaneous interpretation system under test is S2T, the evaluation matrix includes semantic fidelity, end-to-end latency, terminology standardization, output stability, and fluency. When the output modality of the simultaneous interpretation system under test is S2S, the evaluation matrix includes semantic fidelity, end-to-end latency, and terminology standardization.

[0086] In S4 above, when calculating the score of the evaluation unit on each evaluation indicator in the simultaneous interpretation system evaluation matrix, when the evaluation indicator is semantic fidelity, the final semantic similarity score is used as the score corresponding to semantic fidelity, that is... That is, the above This reflects the degree to which the data block preserves the semantics of the source speech.

[0087] When the evaluation metric is end-to-end latency, a preset first exponential decay model is used to map the latency of the evaluation unit into a score, which serves as the score corresponding to the end-to-end latency. The delay time is obtained based on the start time of the speech segments and time-aligned data blocks included in the evaluation unit. The first exponential decay model described above can map the delay time (in seconds) to a score, specifically as shown in the following formula (2):

[0088] (2)

[0089] The meaning of formula (2) above is: when the delay time is 0 seconds, the score is 1; when the delay time is 3 seconds, the score is about 0.22; when the delay time is 5 seconds, the score is less than 0.1. The "0.5" in formula (2) above is the delay attenuation coefficient, which can be adjusted according to the application scenario (this parameter can be increased for scenarios that are sensitive to delay). Formula (2) above is only used as an example with the delay attenuation coefficient set to 0.5. Indicates the delay time.

[0090] When the evaluation metric is terminology standardization, terms are extracted from the time-aligned data blocks included in the evaluation unit, and the matching rate between the extracted terms and terms in a pre-built domain terminology library is calculated. This score is used to determine the terminology standardization score. The aforementioned domain terminology database is pre-built and contains a large number of specialized terms within the domain. The matching rate can be calculated using the following formula (3):

[0091] (3)

[0092] in, This indicates the matching rate. If no terms appear in the above time-aligned data block, the score for this dimension's evaluation metric (i.e., terminology normalization) is 1 (no penalty).

[0093] When the evaluation metric is output stability and the number of evaluation units is at least two, the edit distance change rate between the temporally aligned data blocks included in two adjacent evaluation units is calculated. This edit distance change rate is then mapped to a score using a pre-defined second exponential decay model, which serves as the score corresponding to output stability. The edit distance variation rate is derived based on the length and edit distance of the temporally aligned data blocks in two adjacent evaluation units. The edit distance variation rate between the temporally aligned data blocks in the aforementioned two adjacent evaluation units is... The second exponential decay model described above can map the rate of change of distance into a score, specifically as shown in the following formula (4):

[0094] (4)

[0095] in, Edit distance, also known as Levenshtein distance, refers to the minimum number of single-character edit operations required to transform one string (or text) into another string. In other words, it is the minimum number of single-character edit operations required to transform the time-aligned data block in one evaluation unit into the time-aligned data block in the other evaluation unit. The time score is 1. When the time score is below 0.22, a significant penalty is imposed on the output that changes drastically. The “3” in the above formula (4) is the stability decay coefficient, which is preset. The above formula (4) is only used as an example with the stability decay coefficient set to 3.

[0096] When fluency is the evaluation metric, a pre-trained language model for the target language is used to calculate the translation perplexity of the evaluation unit. The specific calculation method is as follows: the translated text (i.e., the time-order aligned data block) is... Segmented into word sequences according to the word segmenter corresponding to the pre-trained language model of the target language. , This represents the number of lexical units, and the lexical unit sequence is input into a pre-trained language model of the target language to obtain the conditional probability of each lexical unit. Then, the perplexity of the translation is calculated using the following formula (5):

[0097] (5)

[0098] The translation perplexity is linearly mapped to a specified interval to obtain the fluency score, i.e. Specifically, the following formula (6) can be used for calculation:

[0099] (6)

[0100] Among them, when The time score is 1. The time score is 0, and the intermediate linear interpolation is used.

[0101] In step S4 above, when determining the weights of each evaluation indicator, once the number of evaluation units reaches a preset threshold, the entropy value corresponding to each evaluation indicator is calculated based on the score of each evaluation unit on each indicator. The weight of each evaluation indicator is then determined based on its entropy value. Different conference scenarios place varying degrees of emphasis on each dimension's evaluation indicators (e.g., medical conferences prioritize terminology standardization, while business negotiations prioritize delay). An adaptive weight allocation mechanism based on information entropy is introduced to better set the weights of evaluation indicators across different dimensions. As the conference progresses and the cumulative number of evaluation units reaches a preset threshold (e.g., 20), the server automatically calculates the entropy value of the score for each dimension's evaluation indicator. The entropy value reflects the dispersion of the score for that dimension's evaluation indicator. A smaller entropy value indicates a greater difference between different evaluation units, making it more valuable for distinguishing the performance of the simultaneous interpretation system under test, and thus warranting a higher weight. Conversely, a larger entropy value indicates that the scores for that dimension's evaluation indicator tend to be consistent, with low differentiation, and therefore the weight should be reduced. Thus, entropy value and weight are negatively correlated.

[0102] Assuming there is a total The evaluation unit, the first Entropy of evaluation indicators in each dimension The following formula (7) can be used for calculation:

[0103] (7)

[0104] in, For the first The evaluation unit in the first Scores on evaluation metrics across all dimensions.

[0105] Then, the Weights of evaluation indicators in each dimension Determined by the following formula (8):

[0106] (8)

[0107] In this formula (8), the sum of the weights of the evaluation indicators of each dimension is 1, and the weight of the evaluation indicator of the dimension with higher discrimination is greater. The "5" in the formula (8) represents the number of evaluation indicators, which can be set according to the number of evaluation indicators included in the simultaneous interpretation system evaluation matrix. The formula (8) is only an example of the simultaneous interpretation system evaluation matrix including 5 evaluation indicators.

[0108] However, when the number of evaluation units does not reach the preset threshold, the initial weights of the evaluation indicators for each dimension can be preset manually. This can be used in the early stages of the meeting and automatically switched to adaptive mode after sufficient data has been collected.

[0109] The above-mentioned The quality score of each assessment unit is the weighted sum of the scores of the assessment indicators of each dimension, which can be expressed by the following formula (9):

[0110] (9)

[0111] in, Indicates the first The quality score of each evaluation unit, the "5" in the above formula (9) represents the number of evaluation indicators, which can be set according to the number of evaluation indicators included in the above simultaneous interpretation system evaluation matrix. The above formula (9) is only an example of the above simultaneous interpretation system evaluation matrix including 5 evaluation indicators.

[0112] S5: Calculate the overall score of the simultaneous interpretation system under test based on the quality score of the evaluation unit.

[0113] In this specification, the server can calculate the overall score of the simultaneous interpretation system under test based on the quality scores of the evaluation units. The server can directly use the quality score (or the sum of the quality scores of all evaluation units) as the overall score. It can also determine the reliability of the evaluation units based on the standard deviation of each initial score; reliability is negatively correlated with the standard deviation. This is based on a preset exponential decay coefficient and the standard deviation (…). ), determine the overall weight, that is The exponential decay coefficient can be preset; the specific settings are available in the settings provided. That is, the units with high consistency among multiple models contribute more, and the comprehensive score of the simultaneous interpretation system under test is calculated based on the quality score and comprehensive weight of the evaluation units. Specifically, the following formula (10) can be used:

[0114] (10)

[0115] in, This indicates the overall score.

[0116] In some embodiments of this specification, the semantic similarity in S3 above can also be calculated based on each cross-modal semantic similarity model. Each cross-modal semantic similarity model is used to perform the first type and the second type of task. In S3, when calculating semantic similarity, the server can determine the data modality of the temporally aligned data block in the evaluation unit as the first modality, and determine the data modality of the speech segment in the evaluation unit as the second modality. The first modality can be speech or text, and the second modality is speech. Then, the server can determine the first embedding vector of the temporally aligned data block in the evaluation unit through the encoder corresponding to the first modality in each cross-modal semantic similarity model, and determine the second embedding vector of the speech segment in the evaluation unit through the encoder corresponding to the second modality in each cross-modal semantic similarity model. Calculating the semantic similarity between each first embedding vector and each second embedding vector is equivalent to calculating the semantic similarity between the first embedding vector and the second embedding vector obtained based on each cross-modal similarity model.

[0117] Based on this, in S3 above, when calculating the semantic similarity score, the server can map each semantic similarity of the evaluation unit to a specified interval to obtain an initial score. The median of each initial score is then used as the semantic similarity score. Specifically, for the... A cross-modal semantic similarity model, where the initial score is defined as the cosine similarity vector of the embedded vectors. The linear mapping can be calculated using the following formula (11):

[0118] (11)

[0119] in, Cosine similarity (range of values) After transformation : 0 indicates that the semantics of the speech segment and the time-aligned data block in the evaluation unit are completely opposite, and 1 indicates that the semantics of the speech segment and the time-aligned data block in the evaluation unit are completely consistent. Represents the first embedding vector. Represents the second embedding vector. Indicates based on the first The initial scores obtained from the cross-modal semantic similarity model. (The above...) , This indicates the number of cross-modal semantic similarity models.

[0120] Each of the above cross-modal semantic similarity models is calculated independently. Then, the server can use the median of each initial score as the semantic similarity score of the evaluation unit, which can be calculated using the following formula (12):

[0121] (12)

[0122] in, This represents the semantic similarity score. The median was chosen instead of the mean because the median is naturally robust to outliers. Even if a cross-modal semantic similarity model produces extremely low scores due to input modality mismatch, the median remains stable.

[0123] Furthermore, when the output modality of the simultaneous interpretation system under test is S2S and the number of cross-modal semantic similarity models used to perform the second type of task is less than a preset third threshold, the temporally aligned data blocks in the evaluation unit are transcribed into text using a preset automatic speech recognition (ASR) system. The semantic similarity between the speech segments and text in the evaluation unit is determined using the cross-modal semantic similarity models used to perform the first type of task. The aforementioned third threshold is preset and can be specifically set to 3.

[0124] In some embodiments of this specification, the server can also update the evaluation results displayed on the evaluation result interface based on the semantic similarity, latency, score on each evaluation metric, credibility, and the overall score of the simultaneous interpretation system under test of the evaluation unit, and generate an evaluation report based on the evaluation results. The evaluation results include a curve showing the semantic similarity over time, a latency curve (milliseconds), a radar chart of scores on each evaluation metric, a credibility indicator, and a cumulative overall score. The credibility indicator displays the color corresponding to the credibility level; the color of the credibility level differs for different standard deviations within different value ranges. Green and highly reliable. Yellow is reliable. Red indicates low confidence. The above evaluation report can be generated in real time, and the evaluation results are updated immediately after each evaluation unit is processed.

[0125] In some embodiments of this specification, taking Conference A as an example, the speaker delivers Report 1 in English, and the simultaneous interpretation system under test is an S2T output modal simultaneous interpretation system. First, the server can load a domain terminology library (approximately 200 core terms) and select three cross-modal models: BLASER 2.0, SONAR, and WhisperSim. The VAD silence threshold (i.e., the first threshold) is set to 300ms, the latency decay coefficient to 0.5, and the stability decay coefficient to 3. The initial weights of each evaluation metric are initialized: the initial weight for semantic fidelity is set to 0.40, the initial weight for end-to-end latency is set to 0.25, the initial weight for terminology standardization is set to 0.15, the initial weight for output stability is set to 0.10, and the initial weight for fluency is set to 0.10. These initial weights can be adaptively updated subsequently.

[0126] Then, the speaker's source speech stream was acquired in real time at 16kHz, and the source speech stream was divided into frames every 25ms. Each audio frame was timestamped to obtain the first speech. At the same time, text blocks output by the simultaneous interpretation system under test were received via WebSocket (since the output mode of the simultaneous interpretation system under test is S2T, the data blocks are actually text blocks), and each text block carried the start and end timestamps of the corresponding source speech.

[0127] Next, the VAD algorithm is used to detect and segment the first speech, resulting in the first natural semantic window (i.e., speech segment). Seconds. The simultaneous interpretation system under test outputs two text blocks within the time interval corresponding to this natural language processing window. That is... The guidelines now recommend The phrase "SGLT2 inhibitors for all patients with HFrEF" can be used to merge the two text blocks mentioned above to obtain a time-aligned text block. The time coverage range of the merged time-aligned text block is... The alignment quality Δ = 1 - (4.1 - 0.8) / 4.2 ≈ 0.21 < 0.3, indicating successful alignment, meaning it meets the alignment quality requirements. The first natural semantic window and the temporally aligned text block are then merged into one evaluation unit. Delay time. Second.

[0128] Then, multiple cross-modal semantic similarity models are used to calculate the semantic similarity between speech segments and time-aligned text blocks in the evaluation unit, and each semantic similarity is mapped to a specified interval to obtain an initial score. The median of each initial score is then used as the semantic similarity score. The specific calculation results are as follows:

[0129] (1) Model 1 (BLASER 2.0): The initial score corresponding to the semantic similarity between the speech segment and “The guidelines now recommend SGLT2 inhibitors for all patients with HFrEF” was 0.91.

[0130] (2) Model 2 (SONAR): The initial score corresponding to the semantic similarity between the speech segment and “The guidelines now recommend SGLT2 inhibitors for all patients with HFrEF” is 0.94.

[0131] (3) Model 3 (WhisperSim): The initial score corresponding to the semantic similarity between the speech segment and “The guidelines now recommend SGLT2 inhibitors for all patients with HFrEF” is 0.92.

[0132] (4) The median of the above initial scores Standard deviation (Highly reliable).

[0133] Next, based on the semantic similarity score, the score of the evaluation unit on each evaluation indicator in the simultaneous interpretation system evaluation matrix is ​​calculated. A weighted sum is then performed based on the weight and score of each evaluation indicator to obtain the quality score of the evaluation unit. The specific calculation results are as follows:

[0134] (1) Score of semantic fidelity: 0.92.

[0135] (2) Score for end-to-end delay: .

[0136] (3) Terminology standardization: The output contains “SGLT2 inhibitors” and “HFrEF”, which are completely matched with the terminology database, and the score is 1.0.

[0137] (4) Output stability: Since the above evaluation unit is the first unit, the default score is 1.0.

[0138] (5) Fluency: Confusion PPL=18, mapping score (200-18) / 190=0.958.

[0139] (6) Overall score under initial weight: 0.40×0.92+0.25×0.67+0.15×1.0+0.10×1.0+0.10×0.958=0.892.

[0140] Then, as Meeting A progresses, after accumulating 20 evaluation units, the entropy values ​​of the evaluation indicators for each dimension can be calculated. Assuming significant differences in semantic fidelity and terminology standardization (small entropy values), while fluency shows no significant difference (large entropy values), the initial weights for semantic fidelity, terminology standardization, and fluency are automatically adjusted to 0.45, 0.22, and 0.05, respectively. Subsequent evaluation units will then place greater emphasis on the core dimensions in their scoring.

[0141] Finally, the front-end interface (i.e., the evaluation results display interface) displays the overall score change curve, delay curve, radar chart of scores on each evaluation indicator, and credibility indicator (green for high credibility) in real time. A complete evaluation report is automatically generated after Meeting A concludes.

[0142] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A method for evaluating simultaneous interpretation systems based on cross-modal semantic similarity, characterized in that, The method includes: S1: Real-time acquisition of source speech stream, and frame processing and system clock synchronization of the source speech stream to generate audio frames with timestamps, which are combined into the first speech; real-time acquisition of data blocks of the simultaneous interpretation system under test with different output modalities; S2: Semantically segment the first speech to obtain each speech segment, merge the data blocks corresponding to the time intervals of each speech segment to generate a time-aligned data block, and summarize the speech segments that meet the alignment quality requirements and the corresponding time-aligned data blocks into an evaluation unit; S3: Calculate the semantic similarity between the speech segments and time-aligned data blocks in the evaluation unit, and calculate the semantic similarity score based on the semantic similarity of the evaluation unit; S4: Construct a simultaneous interpretation system evaluation matrix. Based on the semantic similarity score, calculate the score of the evaluation unit on each evaluation indicator in the simultaneous interpretation system evaluation matrix. Perform a weighted sum based on the weight and score of each evaluation indicator to obtain the quality score of the evaluation unit. S5: Calculate the overall score of the simultaneous interpretation system under test based on the quality score of the evaluation unit.

2. The method for evaluating a simultaneous interpretation system based on cross-modal semantic similarity as described in claim 1, characterized in that, The output modes of the simultaneous interpretation system under test in S1 include S2T and S2S. When the output mode is S2T, each data block contains text content, the generation timestamp of the text content, and the start and end timestamps of the source speech corresponding to the text content. When the output mode is S2S, each data block contains audio data, the generation timestamp of the audio data, and the start and end timestamps of the source speech corresponding to the audio data.

3. The method for evaluating a simultaneous interpretation system based on cross-modal semantic similarity as described in claim 1, characterized in that, In step S2, the first speech is semantically segmented to obtain each speech segment, which specifically includes: A preset speech activity detection algorithm is used to identify silent segments in the first speech, and silent segments that meet a preset first condition are marked as semantic boundaries. The first condition is that the duration of the silent segment exceeds a preset first threshold. Based on the semantic boundaries, the first speech is semantically divided to obtain speech segments.

4. The method for evaluating a simultaneous interpretation system based on cross-modal semantic similarity as described in claim 1, characterized in that, The method for generating the evaluation unit in S2 is as follows: The time interval corresponding to each speech segment is determined as the first interval, and a data block matching each speech segment is found from each data block and designated as the first data block; the time interval corresponding to the first data block overlaps with the first interval. According to the chronological order, the first data blocks corresponding to each speech segment are merged to obtain the time-aligned data block corresponding to each speech segment. The time coverage interval corresponding to the time-aligned data block is the union of the time intervals of all data blocks in the time-aligned data block. Determine whether the timing alignment data block of each speech segment and each speech segment meet the alignment quality requirements, and summarize the speech segments that meet the alignment quality requirements and their corresponding timing alignment data blocks into an evaluation unit.

5. The method for evaluating a simultaneous interpretation system based on cross-modal semantic similarity as described in claim 4, characterized in that, The determination in S2 of whether the timing alignment data block of each speech segment and each speech segment meet the alignment quality requirements specifically includes: The time coverage interval of the time-aligned data block corresponding to each speech segment is taken as the second interval; Based on the first and second intervals corresponding to each speech segment, a preset alignment cost function is used to calculate the alignment quality of the temporal alignment data blocks corresponding to each speech segment. Determine whether the alignment quality of the temporal alignment data block corresponding to each speech segment and each speech segment meets the alignment quality requirements, wherein the alignment quality requirement is that the alignment quality is not less than a preset second threshold.

6. The method for evaluating a simultaneous interpretation system based on cross-modal semantic similarity as described in claim 1, characterized in that, The semantic similarity in S3 is calculated based on each cross-modal semantic similarity model. Each cross-modal semantic similarity model is used to perform a first type and a second type of task, where the first type is speech to text and the second type is speech to speech. The method for calculating semantic similarity in S3 is as follows: The data mode of the time-aligned data block in the evaluation unit is determined as the first mode, and the data mode of the speech segment in the evaluation unit is determined as the second mode; The first embedding vector of the temporally aligned data block in the evaluation unit is determined by the encoder corresponding to the first modality in each cross-modal semantic similarity model, and the second embedding vector of the speech segment in the evaluation unit is determined by the encoder corresponding to the second modality in each cross-modal semantic similarity model. Calculate the semantic similarity between each first embedding vector and each second embedding vector.

7. The method for evaluating a simultaneous interpretation system based on cross-modal semantic similarity as described in claim 6, characterized in that, The semantic similarity score in S3 is calculated as follows: Each semantic similarity of the evaluation unit is mapped to a specified interval to obtain an initial score; The median of each initial score is used as the semantic similarity score.

8. The method for evaluating a simultaneous interpretation system based on cross-modal semantic similarity as described in claim 6, characterized in that, The method further includes: When the output modality of the simultaneous interpretation system under test is S2S and the number of cross-modal semantic similarity models used to perform the second type of task in each cross-modal semantic similarity model is less than a preset third threshold, the temporally aligned data blocks in the evaluation unit are transcribed into text by a preset automatic speech recognition system. The semantic similarity between the speech segment and the text in the evaluation unit is determined by using the cross-modal semantic similarity model used to perform the first type of task in each of the cross-modal semantic similarity models.

9. The method for evaluating a simultaneous interpretation system based on cross-modal semantic similarity as described in claim 1, characterized in that, The method for constructing the evaluation matrix of the simultaneous interpretation system in S4 is as follows: Based on the output mode of the simultaneous interpretation system under test, the evaluation index corresponding to the simultaneous interpretation system under test is extracted from the preset evaluation index set as the target evaluation index. Based on the aforementioned target evaluation indicators, an evaluation matrix for the simultaneous interpretation system is constructed. When the output mode of the simultaneous interpretation system under test is S2T, the evaluation matrix includes semantic fidelity, end-to-end delay, terminology standardization, output stability, and fluency. When the output mode of the simultaneous interpretation system under test is S2S, the evaluation matrix includes the semantic fidelity, the end-to-end delay, and the terminology standardization.

10. The method for evaluating a simultaneous interpretation system based on cross-modal semantic similarity as described in claim 9, characterized in that, The calculation of the score of the evaluation unit on each evaluation indicator in the simultaneous interpretation system evaluation matrix in step S4 specifically includes: When the evaluation metric is semantic fidelity, the final semantic similarity score is used as the score corresponding to semantic fidelity. When the evaluation metric is the end-to-end delay, a preset first exponential decay model is used to map the delay time of the evaluation unit into a score, which is used as the score corresponding to the end-to-end delay; the delay time is obtained based on the start time of the speech segment and time-aligned data block included in the evaluation unit; When the evaluation metric is terminology standardization, terms are extracted from the time-aligned data blocks included in the evaluation unit, and the matching rate between the extracted terms and the terms in the pre-built domain terminology library is calculated as the score corresponding to the terminology standardization. When the evaluation metric is output stability and the number of evaluation units is at least two, the edit distance change rate between the temporally aligned data blocks included in two adjacent evaluation units is calculated, and the edit distance change rate is mapped to a score through a preset second exponential decay model, which is used as the score corresponding to the output stability; the edit distance change rate is obtained based on the length and edit distance of the temporally aligned data blocks in the two adjacent evaluation units; When the evaluation metric is fluency, a pre-trained language model for the target language is used to calculate the translation perplexity of the evaluation unit, and the translation perplexity is linearly mapped to a specified interval to obtain the score corresponding to the fluency.

11. The method for evaluating a simultaneous interpretation system based on cross-modal semantic similarity as described in claim 1, characterized in that, The method for determining the weights of each evaluation indicator in S4 is as follows: When the number of evaluation units reaches a preset threshold, the entropy value corresponding to each evaluation indicator is calculated based on the score of each evaluation unit on each evaluation indicator. The weight of each evaluation indicator is determined based on the entropy value corresponding to each evaluation indicator, wherein the entropy value is negatively correlated with the weight.

12. The method for evaluating a simultaneous interpretation system based on cross-modal semantic similarity as described in claim 7, characterized in that, S5 specifically includes: The credibility of the evaluation unit is determined based on the standard deviation of each initial score, and the credibility is negatively correlated with the standard deviation. Based on the preset exponential decay coefficient and the standard deviation, the comprehensive weight is determined, and based on the quality score of the evaluation unit and the comprehensive weight, the comprehensive score of the simultaneous interpretation system under test is calculated.

13. The method for evaluating a simultaneous interpretation system based on cross-modal semantic similarity as described in claim 12, characterized in that, The method further includes: Based on the semantic similarity, latency, score on each evaluation indicator, credibility, and overall score of the simultaneous interpretation system under test, the evaluation results in the evaluation result display interface are updated. The evaluation results include a curve of semantic similarity changing over time, a latency curve, a radar chart of scores on each evaluation indicator, a credibility indicator, and a cumulative overall score. The credibility indicator is used to display the color corresponding to the credibility, and the color of the credibility corresponds to different standard deviations within different value ranges. An evaluation report is generated based on the evaluation results.

Images