Performance evaluation methods, electronic devices, and storage media for speaker logging systems

Abstract

This invention discloses a performance evaluation method, electronic device, and storage medium for a speaker log system. Specifically, the method includes: acquiring a reference document and speaker logs output by the speaker log system that match the reference document; extracting a reference paragraph set and a predicted paragraph set corresponding to the reference document and the speaker logs, respectively; generating corresponding first and second subgraphs based on the predicted paragraph set and the reference paragraph set, respectively; assigning edge connections between graph nodes in the first and second subgraphs when paragraph text overlap exists; and evaluating the performance of the speaker log system based on the node connection relationships between the first and second subgraphs. Thus, by constructing a connection subgraph with each paragraph as the smallest unit, the recognition error of short paragraphs is not ignored, resulting in a more comprehensive performance evaluation of the speaker logs.

Description

Technical Field

[0001] This invention belongs to the field of speech processing technology, and particularly relates to a performance evaluation method, electronic device and storage medium for a speaker log system. Background Technology

[0002] Log analytics service providers offer storage, analysis, and even data mining services for structured data. However, for voice logs, in addition to data processing, voice processing technology is essential to unlock their analytical and mining value. Currently, the lack of voice technology support among log analytics service providers is a significant deficiency in voice log analysis.

[0003] With the continuous development of speech logging technology, speaker logs have gained wider attention and application in the industry due to their ability to identify speakers and their speaking duration, thus solving the problem of "who speaks and when." Speaker logs are often used as a preprocessing step in audio tasks, with several application scenarios, such as conferences and telephone recordings. For diary-based datasets, the datasets become more realistic, including spontaneous speeches from different sources. These features cause significant variations in the number of speakers and speech duration, especially for shorter utterances. For log-based methods, joint optimization methods, such as VBx-based methods or end-to-end neural diarization (EEND), and multimodal methods have emerged, can effectively handle overlapping speech and short sentences.

[0004] Currently, the traditional metric for evaluating speaker log systems is DER (Diarization Error Rate), but it doesn't effectively utilize these characteristics for evaluation. DER, on the other hand, provides a comprehensive evaluation of false positives, missed positives, and speaker confusion time relative to the reference time, directly calculating these three types of log-related errors. Due to the intuitiveness of its evaluation metrics, DER is widely used in speaker logs.

[0005] However, DER is insensitive to errors in short segments. This is because short segments naturally occupy less time than long segments, resulting in a greater penalty for errors in long segments. Secondly, `collar` is a temporal option in the log error rate. If `collar` is set to a value greater than 0, periods with `collar` values ​​around segment boundaries will be discarded in the evaluation. This option was originally intended to avoid manually labeled noise near boundaries, but segments shorter than two `collar` values ​​will also be excluded from the evaluation. Therefore, short utterances (e.g., "yes" or "no," with durations less than 1 second) still contain semantic information, and the recognition errors they cause are often ignored. Furthermore, DER ignores errors from people who speak less, failing to achieve a comprehensive and objective evaluation of the speaker log system.

[0006] Currently, the industry has not provided a satisfactory solution to the above problems. Summary of the Invention

[0007] This invention provides a performance evaluation method, electronic device, and storage medium for a speaker log system, which at least solves one of the above-mentioned technical problems.

[0008] In a first aspect, embodiments of the present invention provide a performance evaluation method for a speaker log system, comprising: acquiring a reference document and a speaker log output by the speaker log system that matches the reference document, and extracting a reference paragraph set and a predicted paragraph set corresponding to the reference document and the speaker log, respectively; generating corresponding first subgraphs and second subgraphs based on the predicted paragraph set and the reference paragraph set, wherein each graph node in the first subgraph has a corresponding predicted paragraph, and each graph node in the second subgraph has a corresponding reference paragraph; detecting whether there is corresponding paragraph text overlap between the predicted paragraphs corresponding to each graph node in the first subgraph and the reference paragraphs corresponding to each graph node in the second subgraph, and when paragraph text overlap exists, assigning edge connections between the graph nodes in the first subgraph and the graph nodes in the second subgraph; and performing performance evaluation on the speaker log system based on the node connection relationship between the first subgraph and the second subgraph.

[0009] In a second aspect, embodiments of the present invention provide an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the above-described method.

[0010] Thirdly, embodiments of the present invention provide a storage medium storing one or more programs including execution instructions, the execution instructions being readable and executable by electronic devices (including but not limited to computers, servers, or network devices, etc.) to perform the steps of the method described above.

[0011] Fourthly, embodiments of the present invention also provide a computer program product, the computer program product including a computer program stored on a storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the steps of the above-described method.

[0012] The beneficial effects of the embodiments of the present invention are as follows:

[0013] Based on the speaker log and reference documents, corresponding node subgraphs are generated. The overlap of paragraph text corresponding to aligned nodes in the two subgraphs is detected, and edges are assigned to the corresponding aligned nodes when overlap occurs. A connection subgraph is constructed with each paragraph as the smallest unit, ensuring that errors in short paragraphs are not ignored, resulting in a more comprehensive log performance evaluation. The performance evaluation of the speaker log system utilizes node connection relationships (including connected nodes, isolated nodes, or unconnected nodes), providing direct and clear analysis dimensions and a more efficient evaluation process. Furthermore, since cross-paragraph merging is not required for each speaker, it effectively avoids merging or recognition errors caused by excessively large paragraph gaps, resulting in higher accuracy. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 A flowchart illustrating an example of a performance evaluation method for a speaker log system according to an embodiment of the present invention is shown.

[0016] Figure 2 A flowchart illustrating an example of a performance evaluation method for a speaker log system according to an embodiment of the present invention is shown.

[0017] Figure 3 A schematic diagram of constructing a connection subgraph according to an embodiment of the present invention is shown;

[0018] Figure 4 A flowchart illustrating an example of a performance evaluation method for a speaker log system according to an embodiment of the present invention is shown.

[0019] Figure 5The simulation results comparing the SER and BER indices with traditional indices on various systems are shown.

[0020] Figure 6 This is a schematic diagram of the structure of an embodiment of the electronic device of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0023] This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, elements, data structures, etc., that perform a specific task or implement a specific abstract data type. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0024] In this invention, "module," "system," etc., refer to relevant entities applied to a computer, such as hardware, combinations of hardware and software, software, or software in execution. More specifically, for example, an element can be, but is not limited to, a process running on a processor, a processor, an object, an executable element, an execution thread, a program, and / or a computer. Furthermore, an application program or script running on a server, and the server itself, can also be an element. One or more elements may be in an execution process and / or thread, and elements may be localized on a single computer and / or distributed across two or more computers, and may be run on various computer-readable media. Elements can also communicate via local and / or remote processes based on signals having one or more data packets, for example, signals from data interacting with another element in a local system, a distributed system, and / or interacting with other systems via signals over a network of the Internet.

[0025] Finally, it should be noted that in this document, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0026] It should be noted that DER has almost become the standard metric for evaluating speaker logging systems in the industry. However, the log error rate is no longer adequate for the current situation, especially the paragraph-level error rate. However, the paragraph-level error rate is greatly affected by the model's output resolution, so there is currently no good and balanced method for evaluating speaker logging systems.

[0027] DER is a summary of false alarm, missed speech, and speaker confusion time against a reference time, and it is widely used for speaker amplification. The formula for calculating DER is as follows:

[0028]

[0029] Among them, FA all MS all SC all These represent the total false alarm rate, the false negative rate, and the duration of speaker confusion, respectively. REF all It is the total reference duration. Therefore, the DER metric is the percentage of all error durations divided by the reference duration.

[0030] Besides the DER metric, some industry experts and scholars have innovatively proposed using JER (Jaccard error rate) and CDER (Conversational diarization error rate) to evaluate speaker logs in current related technologies, but some insurmountable problems still exist.

[0031] JER can measure each speaker's errors in a balanced way, but JER still has its limitations. JER still uses time as the main measurement dimension, and for each speaker, there is still the same problem as DER: because long utterances account for most of the duration errors, errors in long utterances will ignore errors in short utterances.

[0032] The JER index is based on speaker-weighted duration UNION. m Error assessment. For each reference speaker m, UNION m It is the speech time of false alarms and misreports, and the speech time of the combined reference speech and hypothetical speech. FA mMS m These represent the false alarm rate and the false negative rate for the reference speaker m, respectively. For each JER i The range is from 0 to 1, representing the degree from all missed matches to perfect matches. Finally, JER is all JERs. m The average value of the JER index ensures that each speaker's errors are treated equally. The formula for calculating the JER index is as follows:

[0033]

[0034] The average value for each speaker can be used to measure a speaker's ability to distinguish JER. However, for each speaker, JER still faces the same problem as DER: errors in longer segments mask errors in shorter segments.

[0035] CDER proposes a metric to mitigate this problem by evaluating from a segment perspective, assessing the speaker log system at the segment level and increasing the weight of short-duration speech. During CDER's calculation, adjacent segments from the same speaker are merged and matched, resulting in the output number of incorrectly identified segments. However, if there are too many speech gaps within a segment, or if the speech intervals are large, the merging operation can lead to inaccurate results or unexpected biases. Furthermore, CDER's fixed IoU threshold strategy allows for higher tolerance for longer segments.

[0036] The CDER indicator is calculated using the following formula:

[0037]

[0038] Here, #error segs represents the number of unmatched segments, and #REF segs represents the total number of segments. However, CDER has two drawbacks: 1) The segment merging operation will lead to inaccurate results if there are too many spoken gaps in the segments; 2) CDER uses a fixed IoU threshold, which is problematic. Different segment lengths should have different IoU sensitivities; for example, the IoU threshold should be larger for longer segments and smaller for shorter segments.

[0039] Figure 1 A flowchart illustrating an example of a performance evaluation method for a speaker log system according to an embodiment of the present invention is shown.

[0040] like Figure 1As shown, in step 110, a reference document and a speaker log output by the speaker log system that matches the reference document are obtained, and the reference paragraph set and predicted paragraph set corresponding to the reference document and speaker log are extracted respectively. Specifically, taking the application of the speaker log system in a speech scenario as an example, the reference document can be a speech manuscript, and the speaker log can be a log output by the speaker log system based on the speaker's speech in the speech manuscript.

[0041] In step 120, a first subgraph and a second subgraph are generated based on the predicted paragraph set and the reference paragraph set, respectively. Here, each graph node in the first subgraph has a corresponding predicted paragraph, and each graph node in the second subgraph has a corresponding reference paragraph. For example, a corresponding first subgraph graph node is generated for each paragraph in the predicted paragraph set, and a corresponding second subgraph graph node is generated for each paragraph in the reference paragraph set.

[0042] In step 130, it is detected whether there is text overlap between the predicted paragraphs corresponding to each first subgraph graph node and the reference paragraphs corresponding to each second subgraph graph node. When text overlap exists, an edge connection is assigned between the first and second subgraph graph nodes. For example, each predicted paragraph can be compared with its corresponding or aligned reference paragraph in turn to determine whether text overlap exists. When text overlap exists, an edge connection is constructed between the graph nodes corresponding to the predicted and reference paragraphs.

[0043] In step 140, the speaker logging system's performance is evaluated based on the node connectivity between the first and second subgraphs. Therefore, by utilizing node connectivity (including connected nodes, isolated nodes, or unconnected nodes) to evaluate the speaker logging system's performance, the analysis dimensions are direct and clear, and the evaluation process is more efficient. Furthermore, compared to CDER's computational evaluation, this embodiment does not require cross-segment merging for speakers, effectively avoiding merging or recognition errors caused by excessively large paragraph gaps, resulting in higher accuracy.

[0044] Through the embodiments of the present invention, corresponding node subgraphs are generated based on the speaker log and reference documents, respectively. The overlap of paragraph text corresponding to the aligned nodes in the two subgraphs is detected, and edge connections are assigned to the corresponding aligned nodes when there is overlap. The connection subgraph is constructed with each paragraph as the smallest unit, which does not ignore the error of short paragraphs, making the log performance evaluation results more comprehensive, thereby helping the business system to obtain the optimal speaker log system.

[0045] Regarding step 140 above, in some embodiments, it is possible to detect whether there are isolated graphical nodes in the second subgraph that are not connected by assigned edges. When an isolated graphical node is detected, it is determined that the speaker log system has a paragraph omission identification error for the reference paragraph corresponding to the isolated graphical node. Thus, based on the isolated graphical nodes in the reference paragraph subgraph, paragraph omission identification errors can be detected quickly.

[0046] Figure 2 A flowchart illustrating an example of a performance evaluation method for a speaker log system according to an embodiment of the present invention is shown.

[0047] like Figure 2 As shown, in step 210, a reference document and a speaker log output by the speaker log system that match the reference document are obtained, and the reference paragraph set and the predicted paragraph set corresponding to the reference document and the speaker log are extracted respectively.

[0048] In step 220, a first subgraph and a second subgraph are generated based on the predicted paragraph set and the reference paragraph set, respectively. Here, each graph node in the first subgraph has a corresponding predicted paragraph, and each graph node in the second subgraph has a corresponding reference paragraph.

[0049] In step 230, speaker reference tags corresponding to each reference paragraph in the reference document are obtained, and speaker prediction tags corresponding to each predicted paragraph in the speaker log are parsed. For example, in a speech scenario, speaker reference tags can represent speaker information corresponding to each reference paragraph in the speech manuscript, and speaker reference tags can represent speaker information corresponding to each predicted paragraph inferred by the speaker system through intelligent algorithms (e.g., machine learning models).

[0050] Furthermore, in some optional implementations, it can also be detected whether the speaker reference label set and the speaker prediction label set match. If they match, it indicates that the speaker log system has no speaker prediction error for the reference document, and the speaker log system has good speaker prediction capabilities. If they do not match, it is determined that the speaker log system has speaker prediction errors. Thus, the performance of the speaker prediction dimension of the speaker log system can be quickly evaluated.

[0051] In step 240, paragraph alignment matching is performed on the predicted paragraph set and the reference paragraph set based on the speaker reference label and the speaker prediction label.

[0052] It should be noted that in some cases, the paragraph numbers between the predicted paragraph set and the reference paragraph set may not be perfectly aligned. For example, if the speaker pauses multiple times while delivering the reference paragraph, one reference paragraph may correspond to multiple predicted paragraphs in the speaker's log.

[0053] Specifically, an optimal matching algorithm can be used to solve the permutation problem in speaker label assignment, distributing speaker reference labels to various predicted segments in the speaker log. For example, the Hungarian algorithm can be used to assign labels. If the predicted speaker number is greater than the reference number, it indicates a label mismatch.

[0054] It should be noted that the speaker label mismatch issue mentioned above was ignored in the evaluation of indicators such as JER, and its error was not calculated. In this embodiment of the invention, the above mismatch issue is detected and recorded as... This represents the error caused by false alarms from the speaker and is recorded in the BER (Balanced Error Rate). The calculation process for BER will be introduced below, where the Speaker... FA This represents the harmonic average of the duration and segment error rate of false alarm speakers. Furthermore, when the system predicts too many candidates, the errors caused by false alarm speakers are not negligible, especially in EEND, where it cannot handle an arbitrary number of speakers well.

[0055] Furthermore, to ensure a one-to-one speaker mapping in subsequent optimal matching, a null hypothesis can be added when the number of referenced speakers exceeds the initial hypothesis. To fill the gaps.

[0056] In step 250, for the paragraph group that has completed paragraph alignment matching, it is detected whether there is paragraph text overlap between the predicted paragraph and the corresponding reference paragraph in the paragraph group, and when paragraph text overlap exists, an edge connection is assigned between the first subgraph graph node and the second subgraph graph node.

[0057] Therefore, specific to the speaker's duration and segmentation error, the duration and segmentation error are calculated for each one-to-one speaker mapping. The overlap of paragraph text is detected for paragraph groups of paragraph alignment matching, satisfying the matching of subgraph nodes of arbitrarily segmented predicted paragraphs, making it suitable for processing speaker logs in various business scenarios.

[0058] Figure 3 A schematic diagram of constructing a connection subgraph according to an embodiment of the present invention is shown.

[0059] It should be noted that, unlike CDER which directly merges adjacent speakers, such as... Figure 3 As shown, a graph-based segment matching strategy was employed to create a graph representing the relationship between reference segments and hypothetical segments, with each node corresponding to a specific segment. Specifically, if there is overlap between the reference and predicted segments, an edge can be specified between their corresponding graph nodes.

[0060] In step 260, for each group of paragraphs corresponding to the first subgraph graph nodes and the second subgraph graph nodes with edge connections, the paragraph text intersection-union ratio between the predicted paragraphs and the corresponding reference paragraphs in the paragraph group is calculated.

[0061] In step 270, when the intersection-over-union ratio (IoU) of the paragraph text is less than the adaptive IoU threshold, it is determined that the speaker log system has an error in recognizing the reference paragraph corresponding to the second subgraph graph node. Here, the adaptive IoU threshold is determined based on the duration length and paragraph number corresponding to the reference paragraph of the second subgraph graph node.

[0062] Specifically, after the connected subgraph is constructed, an IoU matching strategy can be applied to the reference and hypothetical nodes of each connected subgraph. It's important to note that gaps between nodes are not merged. If the IoU is greater than a threshold, the nodes in the connected subgraph are considered to be correctly connected, meaning the reference segment and the hypothetical segment or utterance match; otherwise, the reference segment number in that subgraph is considered to have an identification error. Using the connected subgraph strategy, considering only the number of reference segments, the segment-level error of any hypothetical segmentation can be calculated. Furthermore, isolated nodes without text overlap are also considered to have an identification error.

[0063] It should be noted that, compared to the fixed IoU threshold used in CDER evaluation for IoU matching, this embodiment of the invention employs an adaptive IoU threshold. For example, the IoU threshold is increased for longer utterances and decreased for shorter utterances, satisfying the personalized IoU sensitivity requirements for different segment lengths. Therefore, by constructing a connectivity subgraph and using an adaptive IoU threshold, errors in the prediction results of the reference segment can be accurately and objectively identified.

[0064] Specifically, the IoU matching strategy uses an adaptive IoU threshold, which depends on the reference segment duration (DUR) and number (#NUM). The formula for calculating this threshold is as follows:

[0065]

[0066] Prior information was added to the determination of the adaptive threshold for IoU. The lower bound (lb) in this formula is to prevent excessive short-segment offsets.

[0067] Figure 4 A flowchart illustrating an example of a performance evaluation method for a speaker log system according to an embodiment of the present invention is shown.

[0068] like Figure 4 As shown, in step 401, paragraph-level errors for a specific speaker are calculated.

[0069] Specifically, for each speaker reference tag, the deviation reference segments corresponding to the speaker reference tags that have recognition errors are counted, and the segment-level recognition error of the corresponding speaker reference tag is determined based on the counted deviation reference segments.

[0070] This step is to calculate speaker-related paragraph-level errors. Unlike CDER (Conversation Log Error Rate), which directly merges adjacent segments, this uses a graph-based segment matching strategy. Specifically, a graph is built to define the relationship between reference and hypothesis segments, where nodes are segments or discourse. If there is overlap between the reference and hypothesis, an edge is assigned between them. After the connection graph is constructed, an IoU matching strategy is applied to the reference and hypothesis nodes in each connected subgraph. It's important to note that gaps between nodes are not merged. If the IoU is greater than a threshold, the nodes in this subgraph are considered connected, meaning the reference and hypothesis segments match. Otherwise, the reference segment number in this subgraph is considered incorrect. By using the connected subgraph strategy and basing the error solely on the reference segment number, segment-level errors can be identified for arbitrarily segmented predicted segments.

[0071] Therefore, by comprehensively considering the various deviation reference paragraphs of the speaker reference label, the paragraph-level error corresponding to a specific speaker can be obtained, providing a new performance evaluation dimension for the speaker system.

[0072] In step 403, the time level error for a specific speaker is calculated.

[0073] Specifically, based on speaker logs, the duration of the predicted segment duration for each reference segment in the reference segment set is analyzed. Then, for each reference segment, time-level speech recognition defects in the duration of the predicted segment duration are identified to determine the time-level recognition error corresponding to the reference segment. Here, time-level speech recognition defects include false alarm duration and / or missed alarm duration.

[0074] For example, the time-level identification error corresponding to the reference paragraph can be calculated in the following way:

[0075]

[0076] FA represents the time-level identification error corresponding to the reference paragraph. DUR Indicates the duration of false alarms, MS DUR REF indicates the duration of missed listening. DUR This indicates the duration of the labeled paragraph corresponding to the reference paragraph. Unlike JER, the denominator is the duration of the labels corresponding to each reference paragraph, rather than the joint duration.

[0077] In step 420, the reconciliation error of a particular speaker is calculated.

[0078] It should be understood that since the paragraph-level errors output in step 401 and the time-level errors output in step 403 are both output for the reference paragraph unit, error reconciliation can be performed.

[0079] Specifically, for each speaker reference tag, the time-level recognition error corresponding to each reference paragraph with the speaker reference tag is statistically analyzed to determine the time-level recognition error corresponding to the speaker reference tag. The time-level recognition error and paragraph-level recognition error corresponding to the speaker reference tag are then reconciled to determine the paragraph reconciled recognition error corresponding to the speaker reference tag. Thus, by comprehensively considering the paragraph time-level recognition error and the paragraph-level recognition error through the parameter index of the paragraph reconciled recognition error, the two types of errors are compared together to achieve an effective evaluation of the recognition error of each speaker.

[0080] Here, the speaker-specific duration error is calculated. and piecewise error Then, their harmonic average values ​​(E) can be calculated. s The specific calculation process is as follows:

[0081]

[0082] Compared to the arithmetic mean, the harmonic mean E s It can achieve better results in terms of duration and segmentation error, and also avoids zero error by using eps.

[0083] In step 430, the speaker balance error index is calculated.

[0084] Specifically, the harmonic recognition error corresponding to each speaker reference label is statistically analyzed to determine the speaker balance error index for the speaker log system. For example, the speaker balance error index for the speaker log system is determined by calculating the weighted average of the recognition errors of all speakers. By calculating the error of each speaker and averaging it, the system's recognition error for each speaker is balanced. Therefore, the harmonic average of the time error and paragraph error corresponding to a specific reference segment for each speaker is used, and speaker weighting is applied to obtain the average value, achieving a uniform evaluation of the speech log. The speaker balance error index is used to uniformly evaluate speaker weighted error, duration error, and segmentation error.

[0085] Specifically, speaker-weighted error can be calculated by averaging the specific errors of all speakers:

[0086]

[0087] In step 441, SER (Segment-level error rate) is calculated.

[0088] Here, the SER (Sectional Error Rate) can be determined based on the ratio of the number of error segments in the reference segment to the total number of reference segments. It should be noted that the SER is calculated using the number of error segments in the reference segment, rather than the number of predicted segments, because the reference segment achieves higher accuracy compared to the predicted segment count. Therefore, an improved segment-level error rate (SER) is proposed using connected subgraphs and an adaptive IoU threshold to obtain accurate segment matching.

[0089] SER is determined based on the total number of error segments in the reference segment, rather than using segmented prediction segments, thus ensuring the accuracy of the calculation results. The specific calculation formula is as follows:

[0090]

[0091] In step 443, the BER (Balanced error rate) is calculated.

[0092] Specifically, the BER metric for a speaker log system can be determined by summing the speaker balance error index and the speaker prediction error. Therefore, the BER metric provides a balanced error rate that encompasses all aspects considered by DER, JER, and SER, offering a comprehensive measure for speaker log systems.

[0093] BER is the sum of speaker-weighted average and false positive speaker errors. The specific calculation formula is as follows:

[0094]

[0095] In this embodiment of the invention, taking all these aspects into account, a complete diary evaluation is constituted from time-level errors, paragraph-level errors, and speaker-weighted errors, and speaker logs are evaluated by balancing the error rate BER to cover all aspects considered by DER, JER, and SER.

[0096] In this embodiment of the invention, in order to accurately evaluate the segmentation error, an IoU adaptive strategy is proposed to connect the subgraph for arbitrary segmentation and the offset tolerance to evaluate the segment-level error rate of segmentation faults.

[0097] Furthermore, during the implementation of the innovative solution of this invention, the inventors conducted extensive analysis and experimental verification, analyzing our metrics (e.g., SER and BER) on modular systems, EEND, and real datasets (including audio and audiovisual datasets only). In addition, experiments were conducted on several methods: modular systems, VB-based... x The system and EEND. The results show that BER can provide comprehensive and complete analysis results.

[0098] The following are the relevant details of the experiment:

[0099] I. Experimental Setup:

[0100] All metrics were evaluated using overlapping speech, with an IoU of 0.5 for CDER and an adaptive IoU with a lower bound (lb) IoU of 0.5 for SER and BER. Notably, the collar used in the segment-level metrics (SER and BER) was for segment-level IoU thresholding, not boundary collar. For the evaluation dataset, our metrics were tested on the AMI (hybrid headphones) test set, CALLHOME (LDC2001S97, Disk-8) Part 2, DIHARD II test set, VoxConverse test set, and MSDWild minority test set.

[0101] II. Experimental Procedure:

[0102] 1) Comparison of metrics between different datasets

[0103] First, the metrics proposed in this invention (SER and BER) will be compared with other metrics on several publicly available datasets. Figure 5 The corresponding metric comparison results are shown. All results except for MSDWild(AV) were generated by Pyannote. For Pyannote, a modular pipeline was used: segmentation, an ECAPA-TDNN-based embedding extractor, and aggregated hierarchical clustering (AHC). MSDWild(AV) takes video and audio as input, utilizes a multimodal diagnostic system, and employs the experimental setup described above.

[0104] Compared to other metrics, the bit error rate (BER) takes into account all aspects: speaker weighting, duration, and segmentation errors. For example, in AMI, the JER is low while the BER is high, with a large number of false alarm segments detected. These segments are short in duration but large in number, leading to a high BER and BER. In MSDWild, thanks to visualization, the method can predict speech duration more accurately, reducing DER by 27% and BER by 46%. These examples demonstrate that the newly proposed BER (SER and BER) metrics can provide a comprehensive evaluation for speaker log systems.

[0105] 2) Comparison of metrics across different systems

[0106] The metrics (SER and BER) presented in this embodiment were applied to three binary systems: modular systems, VB, and VB. x And EEND-VC, as shown in the table below:

[0107]

[0108] Table 1

[0109] The modular system comprises oracle VAD, an embedding extractor based on ECAPA-TDNN, and spectral clustering. For fair comparison, both the modular system and VBx use oracle VAD and a VOXCELEB-based training corpus for speaker embedding, without handling overlapping speeches. Compared to the modular system, the VBx generalization algorithm, based on VB-HMM, utilizes an HMM to model speaker transitions and employs variational Bayesian (VB) inference to estimate model parameters. As can be seen from the first and second rows of Table 1, VB... x The method outperforms modular systems in reducing duration error (DER) and segment-level error (SER), which in turn reduces the bit error rate.

[0110] EEND treats speaker logs as a multi-label classification problem and optimizes speaker log labels. To alleviate the problems of long recordings and arbitrary speaker numbers in EEND, EEND-VC first divides the recording into several cassettes (30 seconds in this case). Then, for each cassette, the EEND method trained on Permutation Invariant Training (PIT) is used to generate overlap-aware segmentation. Finally, Vector Clustering (VC) is used to cluster all speech. For the EEND-VC experiments reported in this paper, no prior information, including oracle speaker number and oracle segmentation, was added. Therefore, the DER score is better than VB. xPoorly, for EEND-VC, the DER score is poor, but the SER score is good, as shown in the second and third rows of Table 1. This conflict illustrates that although EEND-VC does not use Oracle segmentation, it is better able to distinguish segments through PIT and "cannot link" constraints, which reduces SER and BER. Additionally, in the second part of BER (Speaker... FA In this context, the balance error caused by a false alarm speaker is a key factor in the EEND-VC modular system and VB. x The difference is significant. This phenomenon is also inconsistent with the fact that the EEND method is inadequate for arbitrary speaker numbers. Through the above analysis, the metrics SER and BER proposed in the embodiments of this invention can evaluate the system from more perspectives.

[0111] In this embodiment, two metrics, SER (Segmentation-Level Error Rate) and BER (Balanced Error Rate), are proposed. Utilizing connected subgraphs and an IoU adaptive strategy, the SER algorithm is proposed to accurately solve the segmentation matching problem under arbitrary segmentation. Based on SER and stimulated by several traditional metrics, BER evaluates speaker weighting, duration, and segmentation error in a unified manner. Experiments demonstrate that BER shows the potential of emerging speaker system algorithms such as multimodal or EEND methods, providing a novel and valuable performance evaluation metric for speaker log systems.

[0112] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of combined actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, as some steps can be performed in other orders or simultaneously according to the present invention. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention. In the above embodiments, the descriptions of each embodiment have their own emphasis; for parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0113] In some embodiments, the present invention provides a non-volatile computer-readable storage medium storing one or more programs including execution instructions, which can be read and executed by electronic devices (including but not limited to computers, servers, or network devices) to perform the performance evaluation method of any of the speaker log systems described above.

[0114] In some embodiments, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions that, when executed by a computer, cause the computer to perform any of the above-described speaker log system performance evaluation methods.

[0115] In some embodiments, the present invention also provides an electronic device comprising: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a performance evaluation method for a speaker log system.

[0116] Figure 6 This is a schematic diagram of the hardware structure of an electronic device for performing a performance evaluation method for a speaker log system according to another embodiment of the present invention, as shown below. Figure 6 As shown, the device includes:

[0117] One or more processors 610 and memory 620, Figure 6 Take the 610 processor as an example.

[0118] The device for performing the performance evaluation method of the speaker log system may further include: an input device 630 and an output device 640.

[0119] The processor 610, memory 620, input device 630, and output device 640 can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0120] The memory 620, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the performance evaluation method of the speaker log system in the embodiments of the present invention. The processor 610 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 620, thereby implementing the performance evaluation method of the speaker log system in the above-described method embodiments.

[0121] The memory 620 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 620 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 620 may optionally include memory remotely located relative to the processor 610, and these remote memories can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0122] Input device 630 can receive input numerical or character information and generate signals related to user settings and function control of the voice interaction device. Output device 640 may include display devices such as a display screen.

[0123] The one or more modules are stored in the memory 620, and when executed by the one or more processors 610, they perform the performance evaluation method of the speaker log system in any of the above method embodiments.

[0124] The above-described product can execute the method provided in the embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in the embodiments of the present invention.

[0125] The electronic devices of this invention exist in various forms, including but not limited to:

[0126] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0127] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include: PDAs, MIDs, and UMPCs, etc.

[0128] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0129] (4) Other airborne electronic devices with data interaction capabilities, such as vehicle-mounted systems installed on vehicles.

[0130] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0131] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A performance evaluation method for a speaker log system, comprising: Obtain a reference document and speaker logs output by a speaker log system that match the reference document, and extract the reference paragraph set and predicted paragraph set corresponding to the reference document and the speaker logs, respectively; Based on the predicted paragraph set and the reference paragraph set, corresponding first subgraphs and second subgraphs are generated respectively, wherein each graphic node in the first subgraph has a corresponding predicted paragraph, and each graphic node in the second subgraph has a corresponding reference paragraph. Detect whether there is text overlap between the predicted paragraph corresponding to each first subgraph graph node and the reference paragraph corresponding to each second subgraph graph node, and when there is text overlap, assign an edge connection between the first subgraph graph node and the second subgraph graph node. The speaker log system is evaluated for performance based on the node connection relationship between the first subgraph and the second subgraph.

2. The method according to claim 1, wherein, The performance evaluation of the speaker log system based on the node connection relationship between the first subgraph and the second subgraph includes: Detect whether there are isolated graph nodes in the second subgraph that are not connected by assigned edges; When the existence of the isolated graphic node is detected, it is determined that the speaker log system has a paragraph omission identification error for the reference paragraph corresponding to the isolated graphic node.

3. The method according to claim 1, wherein, The step of detecting whether there is text overlap between the predicted paragraphs corresponding to each first subgraph graph node and the reference paragraphs corresponding to each second subgraph graph node includes: Obtain the speaker reference tags corresponding to each reference paragraph in the reference document, and parse the speaker prediction tags corresponding to each predicted paragraph in the speaker log; Based on the speaker reference label and the speaker prediction label, paragraph alignment matching is performed on the predicted paragraph set and the reference paragraph set; For a group of paragraphs that have completed paragraph alignment matching, detect whether there is paragraph text overlap between the predicted paragraph and the corresponding reference paragraph in the group.

4. The method according to claim 3, wherein, The performance evaluation of the speaker log system based on the node connection relationship between the first subgraph and the second subgraph includes: For each group of paragraphs corresponding to the first and second subgraph graph nodes with edge connections, calculate the paragraph text intersection-union ratio between the predicted paragraph and the corresponding reference paragraph in the paragraph group; When the intersection-union ratio of the paragraph text is less than the adaptive IoU threshold, it is determined that the speaker log system has an error in recognizing the reference paragraph corresponding to the second subgraph graph node; wherein, the adaptive IoU threshold is determined based on the duration length and paragraph number corresponding to the reference paragraph of the second subgraph graph node.

5. The method according to claim 3, wherein, After obtaining the speaker reference tags corresponding to each reference paragraph in the reference document and parsing the speaker prediction tags corresponding to each predicted paragraph in the speaker log, the method further includes: Detect whether the speaker reference label set and the speaker predicted label set match; When a mismatch occurs, it is determined that the speaker prediction error exists in the speaker log system.

6. The method according to claim 3, wherein, Based on the node connection relationships between the first subgraph and the second subgraph, the speaker logging system is evaluated for performance, including: For each speaker reference tag, the deviation reference segments corresponding to the speaker reference tag with recognition errors are counted, and the segment-level recognition error corresponding to the speaker reference tag is determined based on the counted deviation reference segments.

7. The method according to claim 1 or 6, wherein, After extracting the reference paragraph set and predicted paragraph set corresponding to the reference document and the speaker log, respectively, the method further includes: Based on the speaker log, the predicted duration of each reference paragraph in the reference paragraph set is analyzed; For each of the reference segments, identify time-level speech recognition defects in the duration of the labeled segments corresponding to the reference segments to determine the time-level recognition error corresponding to the reference segments; wherein, the time-level speech recognition defects include false alarm duration and / or missed alarm duration.

8. The method according to claim 7, wherein, The performance evaluation of the speaker log system based on the node connection relationship between the first subgraph and the second subgraph includes: Reference tags for each speaker: The time-level recognition error corresponding to each reference segment with the speaker reference label is statistically analyzed to determine the time-level recognition error corresponding to the speaker reference label. The time-level recognition error and paragraph-level recognition error corresponding to the speaker reference label are harmonized to determine the paragraph harmonization recognition error corresponding to the speaker reference label; The speaker harmonic recognition error corresponding to each speaker reference label is statistically analyzed to determine the speaker balance error index corresponding to the speaker log system.

9. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-8.

10. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-8.

Images