Multi-speaker dialogue voice analysis method and device, equipment and medium

Abstract

The invention relates to the technical field of voice processing, can be applied to business scenes such as financial science and technology and medical health, and discloses a multi-speaker dialogue voice analysis method, device and equipment and a medium, and the method comprises the steps: obtaining a to-be-analyzed multi-speaker dialogue voice, determining a naturalness score based on an acoustic feature, and obtaining a multi-speaker dialogue voice analysis result; determining a semantic consistency score based on voice embedding and semantic embedding corresponding to a preset text, determining a speaker consistency score based on embedding of a plurality of speakers of the same speaker, determining an interaction rationality score based on voice alternate overlapping duration, determining a diversity score based on a variance of voice features, and fusing the scores, the comprehensive mass fraction is obtained. According to the invention, through quantitative evaluation of five dimensions of naturalness, semantic consistency, speaker consistency, interaction rationality and diversity, a comprehensive quality scoring system is established, so that the evaluation result simultaneously reflects voice fluency, content matching degree, identity stability, interaction rhythm rationality and feature richness.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, and in particular to a method, apparatus, device, and medium for analyzing multi-speaker dialogue speech. Background Technology

[0002] With the development of multi-speaker dialogue generation technology, speech synthesis systems are gradually being applied in scenarios such as virtual customer service, intelligent assistants, virtual character dialogues, and film and television dubbing. However, under current technological conditions, there are still significant shortcomings in the quality assessment of multi-speaker dialogues. Traditional assessment methods mainly rely on manual scoring, which is not only time-consuming and labor-intensive but also lacks quantitative standards, making it difficult to guarantee objectivity and consistency. In multi-turn, multi-speaker dialogue scenarios, manual scoring is particularly susceptible to the influence of the evaluator's subjective bias, resulting in assessment results that fail to accurately reflect the true quality of the speech.

[0003] In the fintech sector, voice dialogue systems are widely used in intelligent customer service, risk communication, and business consultation. However, most existing evaluation metrics focus on only a single dimension, such as naturalness of speech or semantic consistency, failing to comprehensively measure the overall quality of multi-speaker dialogues. This limitation makes the system prone to issues such as inconsistent speaker identity switching, unreasonable interaction rhythm, and lack of dialogue diversity when handling complex scenarios, thereby affecting the user experience and reducing customer service efficiency and compliance. The lack of a unified comprehensive scoring system also leads to differences in evaluation standards among different financial institutions, making it difficult to establish comparability and industry-wide applicability.

[0004] In the healthcare field, multi-speaker dialogue voice technology is applied in scenarios such as remote consultations, intelligent diagnostic assistance, and patient follow-up. However, existing technologies also have significant shortcomings: automated evaluation methods typically only cover speech naturalness or some semantic consistency indicators, neglecting the rationality of the alternation between doctors and patients, the processing effect when speech overlaps, and the measurement of diversity dimensions. This deficiency leads to discrepancies between evaluation results and actual doctor-patient communication experiences, thus affecting the application value of dialogue generation systems in clinical assistance. Especially in medical dialogues requiring real-time feedback, existing technologies lack closed-loop feedback mechanisms, failing to correct or optimize the model in real time during speech generation, making it difficult for the system's naturalness and interactivity to meet high standards of medical needs. Summary of the Invention

[0005] The main objective of this invention is to provide a method, apparatus, device, and storage medium for analyzing multi-speaker dialogue speech, aiming to solve the technical problem that existing technologies cannot uniformly quantify and integrate multiple-dimensional quality indicators such as naturalness, semantic consistency, speaker consistency, interaction rationality, and diversity into a comprehensive quality score.

[0006] To achieve the above objectives, the present invention provides a method for analyzing multi-speaker dialogue speech, comprising:

[0007] Acquire the multi-speaker dialogue audio to be analyzed;

[0008] Based on the acoustic features of the multi-speaker dialogue speech, a naturalness score of the multi-speaker dialogue speech is determined.

[0009] Based on the speech embedding of the multi-speaker dialogue speech and the semantic embedding corresponding to the preset text, the semantic consistency score of the multi-speaker dialogue speech is determined.

[0010] Based on the speaker embeddings corresponding to multiple speech segments of the same speaker in the multi-speaker dialogue speech, the speaker consistency score of the multi-speaker dialogue speech is determined.

[0011] Based on the duration of speech alternation and overlap in the multi-speaker dialogue, the interaction rationality score of the multi-speaker dialogue is determined.

[0012] The diversity score of the multi-speaker dialogue is determined based on the variance of the speech features of the multi-speaker dialogue.

[0013] The naturalness score, semantic consistency score, speaker consistency score, interaction rationality score, and diversity score are fused together to obtain the comprehensive quality score of the multi-speaker dialogue speech.

[0014] Furthermore, to achieve the above objectives, the present invention provides an analysis device for multi-speaker dialogue speech, comprising:

[0015] The voice acquisition module is used to acquire the voice of multi-speaker dialogues to be analyzed.

[0016] An acoustic analysis module is used to determine the naturalness score of the multi-speaker dialogue speech based on the acoustic features of the multi-speaker dialogue speech.

[0017] The semantic alignment module is used to determine the semantic consistency score of the multi-speaker dialogue speech based on the speech embedding of the multi-speaker dialogue speech and the semantic embedding corresponding to the preset text.

[0018] The speaker consistency module is used to determine the speaker consistency score of the multi-speaker dialogue speech based on the speaker embeddings corresponding to multiple speech segments of the same speaker in the multi-speaker dialogue speech.

[0019] An interaction analysis module is used to determine the interaction rationality score of the multi-speaker dialogue based on the duration of speech alternation overlap.

[0020] A diversity assessment module is used to determine the diversity score of the multi-speaker dialogue speech based on the variance of the speech features of the multi-speaker dialogue speech.

[0021] The quality fusion module is used to fuse the naturalness score, the semantic consistency score, the speaker consistency score, the interaction rationality score, and the diversity score to obtain the comprehensive quality score of the multi-speaker dialogue speech.

[0022] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a multi-speaker dialogue speech analysis program stored in the memory and executable on the processor, wherein when the multi-speaker dialogue speech analysis program is executed by the processor, it implements the steps of the multi-speaker dialogue speech analysis method as described above.

[0023] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a multi-speaker dialogue speech analysis program, which, when executed by a processor, implements the steps of the multi-speaker dialogue speech analysis method described above.

[0024] Beneficial Effects: This invention relates to the field of speech processing technology and can be applied to business scenarios such as fintech and healthcare. It discloses a method, apparatus, device, and medium for analyzing multi-speaker dialogue speech, including: acquiring the multi-speaker dialogue speech to be analyzed; determining a naturalness score based on the acoustic features of the speech; determining a semantic consistency score based on the semantic embedding of the speech and the corresponding semantic embedding of the preset text; determining a speaker consistency score based on the embedding of multiple speakers of the same speaker; determining an interaction rationality score based on the duration of speech alternation and overlap; determining a diversity score based on the variance of the speech features; and fusing the naturalness score, semantic consistency score, speaker consistency score, interaction rationality score, and diversity score to obtain a comprehensive quality score for the multi-speaker dialogue speech. This invention quantifies multiple dimensions of indicators such as speech naturalness, semantic consistency, speaker consistency, interaction rationality, and diversity to construct a unified comprehensive quality scoring system, enabling the evaluation results to comprehensively reflect the natural fluency, content matching degree, identity stability, interaction rhythm rationality, and feature richness of multi-speaker dialogue speech. By integrating multi-dimensional indicators, the one-sidedness caused by single-dimensional evaluation is avoided, and an objective, automated and comparable comprehensive evaluation of the speech quality of multi-speaker dialogues is achieved. Attached Figure Description

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0026] Figure 1This is a schematic diagram of an application environment for a multi-speaker dialogue speech analysis method according to an embodiment of the present invention;

[0027] Figure 2 This is a flowchart illustrating an embodiment of the multi-speaker dialogue speech analysis method of the present invention;

[0028] Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the multi-speaker dialogue speech analysis device of the present invention;

[0029] Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0030] Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0031] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0032] The method for analyzing multi-speaker dialogue speech provided in this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can obtain the multi-speaker dialogue speech to be analyzed from the client, determine a naturalness score based on the acoustic features of the speech, a semantic consistency score based on the semantic embedding of the speech and the corresponding semantic embedding of the preset text, a speaker consistency score based on the multiple speaker embeddings of the same speaker, an interaction rationality score based on the duration of speech alternation and overlap, and a diversity score based on the variance of the speech features. The naturalness score, semantic consistency score, speaker consistency score, interaction rationality score, and diversity score are then integrated to obtain a comprehensive quality score for the multi-speaker dialogue speech. This invention quantifies multiple dimensions of indicators such as speech naturalness, semantic consistency, speaker consistency, interaction rationality, and diversity to construct a unified comprehensive quality scoring system. This allows the evaluation results to comprehensively reflect the naturalness and fluency, content matching, identity stability, interaction rhythm rationality, and feature richness of the multi-speaker dialogue speech. By integrating multiple dimensions of indicators, the one-sidedness caused by single-dimensional evaluation is avoided, achieving an objective, automated, and comparable comprehensive evaluation of the quality of multi-speaker dialogue speech. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0033] Please see Figure 2 , Figure 2This is a flowchart illustrating an embodiment of the multi-speaker dialogue speech analysis method provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0034] like Figure 2 As shown, the method for analyzing multi-speaker dialogue speech proposed in this invention includes the following steps:

[0035] S10, acquire the multi-speaker dialogue voice to be analyzed;

[0036] In this embodiment, acquiring multi-speaker dialogue audio provides reliable input for subsequent evaluation stages. The goal is to create data assets that can be directly incorporated into calculations for naturalness, semantic consistency, speaker consistency, interaction rationality, and diversity, while ensuring complete recording of time references, vocal tract structure, speaker distinguishability, and environmental information. Multi-speaker dialogue audio refers to a collection of audio signals generated by the continuous or alternating speech of at least two participants in the same conversation. It can originate from on-site recordings, captured network calls, retrieved from streaming media, or imported from historical corpora. Acquisition encompasses continuous stages including collection, synchronization, cleansing, standardization, indexing, and storage, and is constrained by explicit metadata descriptions to ensure reproducibility in subsequent stages.

[0037] First, the acquisition method and signal path are clearly defined. In on-site recording scenarios, microphone arrays, distributed multi-point pickup, or stereo devices are used to form a multi-channel audio stream. A one-to-one mapping is established between channels and physical sensors, and the array geometry, microphone model, sensitivity, gain value, and room acoustic parameters are recorded to provide spatiotemporal reference. In network call scenarios, transport layer data is accessed, and jitter buffering, packet loss concealment, and clock recovery mechanisms are established to ensure audio frame continuity and sampling point stability. When importing historical data, files from different sources are uniformly batch-verified to clarify the original sampling rate, quantization accuracy, encoding format, number of channels, and duration. A consistent session numbering and file naming rule is established across the database to form a traceable data list.

[0038] Secondly, a unified time base and channel synchronization are established. A single clock is used as the session time reference, and all channels are aligned to the same sampling start point and sampling period. The alignment relationship between the starting sampling index and UTC is recorded to eliminate sampling drift introduced by different devices at the acquisition end. Microsecond-level deviation measurements are performed on each channel using cross-correlation delay estimation or calibration pulses, generating synchronization correction values ​​and writing them into metadata to ensure consistency of timing events, facilitating subsequent interaction overlap calculations and speaker handover detection. To avoid long-term session accumulation errors, a segmented resynchronization strategy is introduced, updating the delay estimate with a fixed window period and performing local resampling correction to ensure that the synchronization error of the entire session is under control.

[0039] Then, environmental purification and usability enhancement are performed. Bandpass filtering, DC bias removal, and amplitude safety limiting are applied to the input stream to avoid saturation distortion. In scenarios with echo paths, a reference signal path is connected, and echo cancellation is enabled to reduce far-end leakage. Noise suppression can employ spectral masking, time-frequency subband estimation, or end-to-end deep networks to output target signal and residual noise power estimates for subsequent confidence calculation and weighting in the scoring process. In the case of an array configuration, beamforming weights can be calculated based on array geometry and incident direction to generate enhanced streams pointing to different speaker locations, while preserving the original channels for auditing and reprocessing.

[0040] After purification, format standardization is performed. Sampling rate and quantization accuracy are unified, and lossless encoding or pulse code modulation is used for storage. Dithering strategies are applied to suppress quantization errors, and deviations between the declared sampling rate and the actual sampling points are detected and corrected. The standardized file segmentation strategy uses sessions as the aggregation unit and audio segments as the smallest granularity, ensuring no cross-session splicing. Simultaneously, the start and end timestamps of segmentation and the reason for segmentation are recorded to ensure subsequent location back to any original segment.

[0041] Next, indexing and structured description are performed. Based on the timestamps of speech segments output by speech activity detection, a unique identifier is generated for each segment, recording the channel source, energy statistics, signal-to-noise ratio estimation, and preliminary confidence level. If a speaker identifier already exists from the upstream system or registry, it is mapped to the segment and written into the metadata. If not, a separation or separation plus clustering process can be run to generate temporary speaker labels. The labels are aligned with the segment time and continuously numbered, and can be replaced with stable identifiers later through an external mapping table. All metadata is stored in a structured document, with fields including session number, time base, channel mapping, sampling and quantization specifications, device and scene parameters, segment list, speaker labels, quality diagnostic indicators, and checksums for consistency checks and process tracking.

[0042] Finally, consistency verification and secure storage are completed. Checksum comparisons, duration consistency checks, sample point and timestamp consistency checks, channel order consistency checks, and segment coverage checks are performed on audio files and metadata. Data security is protected through encryption and access control, employing a session-level directory structure and index list, supporting retrieval by session number, time range, speaker tag, and quality threshold. When interfacing with downstream evaluation interfaces, list paths, relative file addresses, timestamp units, floating-point units, and placeholder conventions for missing fields are provided to avoid interface ambiguity.

[0043] Time synchronization and clock recovery originate from multimedia transmission and array signal processing; beamforming from array directional enhancement; echo cancellation from dual-talk communication engineering; speech activity detection and separation from speech signal processing and pattern recognition; and metadata organization from corpus engineering and traceable data governance. For example, speaker identifiers can come from anonymous IDs assigned by the registration system, session access port mappings, service-side annotations, or automatic clustering labels; timestamps can be achieved using sampling point counting or UTC millisecond timing; noise estimation can be achieved through statistical modeling or learning networks, both within the scope of acquisition. To support subsequent scoring processes, quality diagnostic fields remain open, allowing the addition of descriptors such as frame drop rate, peak factor, and short-time energy distribution, without limiting the specific set of fields.

[0044] For real-time conferencing scenarios, an eight-channel circular array is used for audio pickup, with the conversation device placed at the center of the array. The acquisition end is driven by a unified sampling clock, and multiple channels are synchronously written to the audio buffer. During the meeting, a fixed-length sliding window is used to perform cross-correlation delay estimation, obtaining the minute offset of each channel pair and correcting it using phase compensation. In noisy office environments, a time-frequency masking network is first applied to output a speech mask, which is then superimposed with bandpass filtering and energy normalization. Subsequently, beamforming weights are calculated based on the target speaker's location to generate a directional enhancement stream. Speech activity detection outputs segment timestamps, which are superimposed with energy thresholds for secondary confirmation, resulting in a segment list. If the service side provides a participant login ID, the channel source and login ID are written into a mapping table at the access layer, and the segment directly references this ID; otherwise, embedding extraction and clustering are performed to generate temporary labels. All results are stored in a conversation list file aligned with the audio file. The list includes the sampling rate, quantization precision, conversation start and end times, channel name, segment timestamp sequence, and label sequence.

[0045] For voice customer service playback scenarios, the input consists of dual network voice streams, representing the client and agent respectively. After decapsulation, the signals enter a jitter buffer and packet loss hiding process to maintain frame-level continuity. An echo reference path is established between the two signals, and echo cancellation is performed to reduce far-end leakage and avoid interference from false overlap in subsequent interaction overlap calculations. Unified resampling and quantization are performed to the target specification, and the consistency between the declared sampling rate and frame duration is verified. A time base and session number are generated by combining call signaling. After segment detection, one side is labeled as the client and the other as the agent. If a channel switch occurs, the switch time point is recorded in the list and the labels are automatically adjusted. The quality diagnosis module records the proportion of silent segments, peak percentage, dynamic range, and frame drop rate, and provides alarm indicators for sessions exceeding the threshold, facilitating the selection of filtering strategies in subsequent evaluation stages.

[0046] For mobile terminal data acquisition scenarios, a dual-microphone device is used, and a near-end speech priority strategy is enabled. The uplink channel is configured with automatic gain control and amplitude limiting protection to prevent near-end popping; the far-end playback is set as a reference input to enhance echo cancellation. The device generates a high-precision monotonic clock, periodically synchronized with network time, and records the synchronization error curve, providing it in the inventory. In walking or outdoor scenarios, wind noise and impulse noise are frequent; transient detection and adaptive thresholding are added to improve the noise reduction effect. Sections judged as low-quality are marked with low confidence, allowing for reduced weighting in subsequent evaluations. To save bandwidth, in-situ compression and delayed upload are supported; compression and decompression parameters are recorded in the inventory, and the evaluation end restores the data to a standard format before downstream processing.

[0047] For historical corpus integration scenarios, files from different sources are read in batches, uniformly converted to the target specification, and indexed using conversations as the organizational unit. If existing annotation files exist, such as time annotations or speaker annotations, format adaptation and time benchmark alignment are performed, linearly mapping the annotations to the new timeline, and the original mapping relationship is saved for traceability. For samples with inconsistent sampling rates or significant time drift, a segmented resynchronization strategy is adopted to reconstruct accurate timestamps. All conversion and correction operations are written to the processing log, and the log path is referenced in the manifest for easy auditing.

[0048] For different scenarios, parameter adjustments and optimizations can be made. The sampling rate and quantization accuracy can be adjusted according to downstream evaluation requirements. Real-time scenarios tend to use lower sampling to reduce bandwidth, while offline high-fidelity evaluation can use higher sampling to retain high-frequency details. The delay estimation window length and update cycle can be adjusted according to the session duration and device stability. A longer window is used in conference scenarios to enhance robustness, while a shorter window is used in mobile scenarios to improve tracking capabilities. The noise suppression intensity and threshold can be adaptively adjusted according to the ambient noise spectral density to maintain a balance between speech intelligibility and fidelity. The echo cancellation filter length and step size can be dynamically adjusted according to the dual-talk ratio to reduce dual-talk suppression errors.

[0049] This embodiment aligns multi-source speech data across time references, channel structures, and format specifications through a unified acquisition and standardization process. Combined with segment timestamps and speaker tags, it forms searchable conversational data assets, significantly reducing evaluation biases introduced by acquisition differences and time drift. Noise suppression, echo cancellation, and beamforming enhance speech intelligibility and signal-to-noise ratio, providing stable input for naturalness and semantic consistency calculations. Accurate timestamps and channel synchronization ensure that interaction overlap duration calculations are not affected by false overlaps. Structured metadata and quality diagnostics provide weighting criteria for speaker consistency and diversity assessments. This achieves a reliable, clearly defined, traceable, and configurable acquisition process, supporting the objective and comparable output of subsequent comprehensive quality scores.

[0050] S20, Based on the acoustic features of the multi-speaker dialogue speech, determine the naturalness score of the multi-speaker dialogue speech;

[0051] In this embodiment, the naturalness score is determined based on the acoustic features of multi-speaker dialogue speech. The core of this method lies in modeling the physical and perceptual features of the speech signal and mapping them to quantifiable fluency metrics. Acoustic features typically refer to the set of parameters extracted from the processed speech waveform, encompassing spectral envelope, formant positions, fundamental frequency trajectory, energy distribution, duration characteristics, spectral tilt, jitter, and tremolo. These features originate from signal processing and phonetics research. For example, Mel-frequency cepstral coefficients are often used to describe the spectral shape consistent with auditory perception, linear prediction coefficients are used to describe the vocal tract model, and the fundamental frequency trajectory directly reflects the naturalness of intonation. After acquiring the multi-speaker speech, a short-time Fourier transform is performed on the signal, dividing the time-domain signal into frames with a window length of twenty to thirty milliseconds. Edge effects are suppressed using a Hamming window to obtain the spectrum. The spectrum is further mapped to the Mel scale to obtain the Mel-frequency cepstral coefficients. To compensate for differences caused by different recording conditions, cepstral mean normalization and energy normalization are also performed to ensure comparability between different environments and speakers.

[0052] The determination of naturalness scores relies not only on static features but also on dynamic temporal patterns. To this end, a standardized feature sequence is used as input to construct a feature vector containing temporal context. This vector is layered with several frames to form a context window, and modeled using a Long Short-Term Memory (LSTM) network or a convolutional temporal network to capture speech rhythm and pronunciation smoothness. The modeling goal is to assess whether there are abrupt breaks in pronunciation continuity, whether the intonation is natural, and whether the energy and duration conform to human speech patterns. This feature vector is then input into a pre-trained naturalness scoring model, typically trained on large-scale manually labeled data, with annotations derived from subjective listening scores, such as MOS (Mean Opinion Score). The model outputs a real-valued value representing the naturalness and fluency of a speech segment.

[0053] Since the naturalness score directly output by the model may exhibit short-term fluctuations, further smoothing is required. Common methods include moving average and low-pass filtering. Moving average eliminates local jitter by taking the mean within a fixed window, while low-pass filtering smooths the output trajectory by filtering out high-frequency components, making the naturalness score more stable in the temporal dimension. The smoothed naturalness score is the final output and can be used as one of the inputs for subsequent fusion steps.

[0054] In practical implementation, different combinations of acoustic features and modeling methods can be employed. For example, a combination of Mel frequency cepstral coefficients and fundamental frequency trajectories can be used to emphasize acoustic stability; alternatively, log-Mel spectrum and duration-normalized features can be used to emphasize pronunciation rhythm. In terms of modeling, a regression model based on deep neural networks can be used to directly predict the naturalness score; alternatively, contrastive learning can be used to distinguish between high-naturalness and low-naturalness samples, thereby obtaining a more robust scoring model. Parameters can be adjusted for different scenarios; for example, spectral subtraction or neural network noise reduction can be added in high-noise scenarios, and feature normalization intensity can be enhanced in cross-device scenarios to eliminate device differences.

[0055] This embodiment extracts acoustic features and maps them to naturalness scores, enabling the quantification of speech fluency and pronunciation rationality, overcoming the problems of non-repeatability and low efficiency in traditional subjective scoring. Due to the use of temporal modeling and smoothing mechanisms, the results retain detail sensitivity while avoiding excessive fluctuations, thus providing stable input for subsequent semantic consistency and interaction rationality analysis in multi-speaker scenarios.

[0056] S30, based on the speech embedding of the multi-speaker dialogue speech and the semantic embedding corresponding to the preset text, determine the semantic consistency score of the multi-speaker dialogue speech;

[0057] In this embodiment, the semantic consistency score is determined based on the speech embedding of multi-speaker dialogue and the semantic embedding corresponding to the preset text. The core is to convert the speech signal into a high-dimensional vector and establish a correspondence with the corresponding text semantic representation to measure the degree of semantic fit between the two. Speech embedding is a vectorized representation extracted by inputting the audio signal into an acoustic coding network. Common implementations include using end-to-end models based on convolutional neural networks or Transformers. These embeddings can capture the prosody, articulation, and temporal information of speech. Semantic embedding originates from the processing of the preset text and is usually obtained through pre-trained language models such as BERT and RoBERTa, which can effectively represent the distribution features of the text semantic space.

[0058] The first step in the processing is to establish the alignment between the speech embedding sequence and the semantic embedding sequence. Alignment can be achieved using dynamic time warping, which matches speech frames with text units in time, ensuring a clear correspondence between each speech segment and its corresponding text in multi-speaker scenarios. After alignment, both the speech and semantic embeddings are normalized to eliminate scale differences and improve the stability of similarity calculation. The normalized vectors are then used to obtain a similarity score sequence through frame-by-frame dot product operations; a higher dot product value indicates a better semantic fit between the speech and text.

[0059] Because speech data may contain invalid segments, such as silent regions or low-confidence recognition parts, it is necessary to filter the similarity score sequence based on the speech activity detection results, removing portions below a threshold and retaining only valid scores. Furthermore, to reflect the importance of speech energy in semantic matching, dynamic weights need to be assigned to the similarity scores based on the energy intensity of each speech frame; regions with higher weights often correspond to areas with clear pronunciation and high information content. Finally, a semantic consistency score is obtained through a dynamic weighted average, achieving a quantitative measurement of the semantic relationship between the speech content and the pre-defined text.

[0060] In implementation, different speech embedding extraction networks can be employed. For example, an x-vector-based speaker recognition model can be used to generate embeddings, highlighting speaker independence, while a self-supervised model like Wav2Vec2.0 can be used to capture finer-grained speech and semantic features. Semantic embedding can be obtained through a general language model or a domain-specific model, such as using a model trained on medical corpora for medical dialogues or a model trained on financial domain corpora for financial customer service dialogues, to improve matching accuracy. Regarding alignment methods, in addition to dynamic time warping, attention alignment mechanisms can be used to enhance long sequence processing capabilities. Weight allocation can also be adjusted according to the scenario; for example, shortening the weighting window in fast-paced dialogues and introducing an exponential decay mechanism to highlight recent statements in long dialogues.

[0061] This embodiment effectively measures the consistency between speech expression and text semantics through the alignment and weighted matching of speech embedding and semantic embedding, avoiding biases caused by relying solely on speech recognition results. This process enables automated semantic fit analysis in multi-speaker scenarios, overcoming the inefficiencies and non-reproducible results of manual subjective judgment, and providing reliable semantic dimension input for calculating the overall quality score.

[0062] S40, Based on the speaker embeddings corresponding to multiple speech segments of the same speaker in the multi-speaker dialogue speech, determine the speaker consistency score of the multi-speaker dialogue speech;

[0063] In this embodiment, during the analysis of multi-speaker dialogue speech, it is necessary to determine whether the same speaker maintains identity continuity across different speech segments. This involves determining the speaker consistency score based on multiple speaker embeddings of the same speaker. Speaker embeddings are high-dimensional vectors obtained by extracting voiceprint features from speech segments. They reflect an individual's vocal tract characteristics, timbre features, and pronunciation habits. Extraction methods typically rely on deep neural networks, such as the x-vector model or the ECAPA-TDNN model. These models can map speech segments of varying durations into a fixed-length vector space of uniform dimension.

[0064] To ensure correct identification of segments from the same speaker, it is necessary to first determine the speaker's identity for each speech segment using a speaker separation or segmentation recognition module, and then extract the corresponding embedding vectors for speech segments belonging to the same speaker. These embedding vectors are sorted according to timestamps to form a speaker embedding time series, ensuring the sequential nature of the speech segments. By constructing such a time series, the stability and continuity of a speaker's pronunciation in a dialogue can be effectively analyzed.

[0065] In the specific calculation, cosine similarity is typically calculated for the embedding vectors of adjacent speech segments to generate a temporal similarity matrix. If the cosine similarity value is stable and high, it indicates that the speaker's identity characteristics remain consistent throughout the dialogue; if the similarity value fluctuates greatly or drops abruptly, there may be speaker switching or instability in the synthesized speech. Finally, these similarity indicators are normalized and weighted to obtain a speaker consistency score, which is used to quantify the speaker's stability in multi-turn dialogues.

[0066] In terms of implementation, different speaker embedding extraction methods can be employed. For example, the x-vector model can be used to capture global speaker features, while the ECAPA-TDNN model based on convolutional neural networks can be used to enhance the representation of local acoustic information. Regarding temporal sorting, ascending order can be based on the starting timestamp generated by the speech recognition system, or time correction can be performed using alignment algorithms to avoid incorrect segmentation of long speech segments. In addition to cosine similarity, Mahalanobis distance or KL divergence can be introduced to enhance discriminative power in multi-speaker scenarios. For long dialogues, a sliding window mechanism can be introduced to calculate similarity only within a local timeframe, reducing computational complexity and highlighting the impact of recent speeches.

[0067] This embodiment effectively measures the continuity of a speaker's acoustic features in long dialogues by extracting multiple embeddings of the same speaker and constructing a time series, avoiding identity inconsistencies caused by synthesized speech or recognition errors. This approach allows speaker consistency scores to reflect the stability and authenticity of speech performance in dialogues, thus providing a reliable basis for comprehensive quality assessment.

[0068] S50, Based on the duration of speech alternation and overlap in the multi-speaker dialogue, determine the interaction rationality score of the multi-speaker dialogue.

[0069] In this embodiment, speech alternation and speech overlap are important indicators of the rationality of interaction in multi-speaker dialogues. Speech alternation refers to one speaker finishing speaking and another immediately following, reflecting the rhythm and cooperation of the dialogue. Speech overlap occurs when two or more speakers speak within the same time period, which can be a natural, short interruption or response, or an unreasonable, excessive interference. To quantify this rationality of interaction, it is necessary to calculate based on the duration of speech alternation and overlap in the multi-speaker dialogue.

[0070] First, the entire speech stream needs to be time-segmented to obtain the boundaries of each speaker's speech segments. Boundary information includes start and end timestamps. This information can be obtained using Voice Activity Detection (VAD) and Speaker Diarization techniques. Using this boundary information, the alternation points and overlapping regions between two speakers can be identified.

[0071] During the calculation, for alternating scenarios, it's necessary to detect the interval between the end time of one speech segment and the start time of the next. If the interval is too long, it may indicate excessive pauses and an unnatural rhythm; if the interval is too short or even overlaps, further analysis of its rationality is required. For overlapping scenarios, the duration of the overlapping segments needs to be calculated directly. All overlapping durations are then accumulated to obtain the total overlapping time of the entire dialogue.

[0072] To ensure the evaluation is meaningful, reference dialogue samples or pre-defined interaction models need to be introduced to provide a reasonable distribution of alternation and overlap durations across different scenarios. For example, in customer service conversations, moderate overlap might involve agreeable interjections like "um" or "okay," typically lasting less than a second; in academic discussions, longer overlaps are more common. By comparing the actual overlap duration with the expected duration of the reference model, the time difference can be calculated, and the time synchronization error rate can be further obtained. Finally, the time synchronization error rate is combined with a pre-defined interaction rhythm threshold to map it into a score used to quantify the reasonableness of the interaction.

[0073] The computation of interaction legitimacy can be achieved in different ways. One approach is to employ a speech activity detection method based on a Hidden Markov Model (HMM) to ensure accurate segmentation. Another approach is to utilize end-to-end segmentation models based on deep learning, such as a combined model of convolutional neural networks and bidirectional long short-term memory networks, to improve the robustness of overlapping segment detection.

[0074] For timestamp alignment, a generalized cross-correlation phase transform delay estimation method can be used to improve the accuracy of speech boundary alignment. The reasonableness threshold can be set by training the system with annotated corpora from dialogues across different domains, allowing it to adaptively adjust in scenarios such as fintech customer service and healthcare consultation.

[0075] When calculating the reasonableness score of an interaction, different weighting methods can be chosen. For example, a lower penalty weight can be assigned to short-term natural overlaps, and a higher penalty weight can be assigned to long-term unreasonable overlaps. A penalty function for alternation intervals can also be introduced to reduce the score for excessively long pauses.

[0076] This embodiment, by calculating the duration of speech alternation and overlap, can comprehensively reflect the naturalness of interaction in multi-speaker dialogues. This method can identify problems such as excessive pauses or unreasonable interruptions, thus providing a quantified interaction rationality score. This score not only helps determine whether the dialogue is natural and fluent but also provides feedback to speech generation or synthesis systems, improving the rhythm and comprehensibility of the interaction.

[0077] S60, Based on the variance of the speech features of the multi-speaker dialogue speech, determine the diversity score of the multi-speaker dialogue speech;

[0078] In this embodiment, in multi-speaker dialogues, the acoustic features and speech segments of different speakers differ, and these differences directly determine the richness and variability of the dialogue. To quantify this variation, the variance of speech features is used to calculate a diversity score. Variance reflects the degree of dispersion in the data distribution; a larger value indicates more significant differences in speech features, thus representing a higher level of diversity.

[0079] The first step is to extract the acoustic features of each speaker from the spoken dialogue. Common features include Mel-frequency cepstral coefficients (MFCC), perceptual linear prediction (PLP) features, spectral envelope, fundamental frequency trajectory, and energy distribution. These features can be calculated using short-time Fourier transform and filter banks, and constructed as a multidimensional feature vector sequence.

[0080] Next, the variance of each dimension in the multidimensional feature vector needs to be calculated individually. The variance value of each dimension reflects the distribution and dispersion of that feature throughout the entire dialogue speech. For example, a larger fundamental frequency variance indicates rich intonation variations in the speaker; a larger energy variance indicates significant differences in vocal intensity. By summing the variances of each dimension, a feature variance vector can be generated.

[0081] To avoid the impact of differences in the numerical ranges of different feature dimensions, it is necessary to perform a normalization operation on the feature variance vector. This is typically achieved using extreme value normalization or standard deviation normalization methods to map all dimensions to the same range. The normalized vector ensures that the features are comparable in subsequent weighted calculations.

[0082] Finally, the normalized variance vector needs to be weighted and fused according to preset weight coefficients to obtain a comprehensive variance value. The weight coefficients can be configured according to different scenarios; for example, in emotional interaction scenarios, fundamental frequency variance and energy variance have higher weights; in task-oriented dialogue scenarios, speech rate-related features may have greater weights. The comprehensive variance value is then compared with a preset threshold range, mapped to a diversity score, which is ultimately used to quantify the degree of change in the entire dialogue.

[0083] Different implementation methods can be used to calculate diversity scores. One approach directly extracts traditional acoustic features such as MFCC and energy features, and then uses matrix operations to batch calculate variance. Another approach uses the intermediate layer outputs of a deep learning model as speech features, for example, using the embedding layer vectors of a self-supervised pre-trained model, and then calculates the variance to measure diversity. Furthermore, the emotion embeddings and prosodic embeddings of speech segments can be incorporated into the variance calculation, allowing the diversity score to reflect not only acoustic differences but also the richness of emotion and interaction style.

[0084] Weighting coefficients can be set manually or learned automatically through data-driven methods. For example, training a model on a financial customer service corpus can determine which feature dimensions are most sensitive to the diversity of customer service interactions; training a model on a medical consultation corpus may reveal that emotion-related features contribute more to diversity.

[0085] For threshold ranges, a distribution baseline can be obtained by statistically analyzing large-scale dialogue data in different scenarios. For example, in normal customer service conversations, diversity scores are usually concentrated at a moderate level; in film and television dubbing scenarios, scores may be higher because the changes in emotion and tone are more dramatic.

[0086] This embodiment introduces speech feature variance as a calculation basis to quantify the variability and richness of multi-speaker dialogues. This method can effectively identify monotonous dialogues and reflect changes in tone, energy, and emotion. Combining normalization and weighting mechanisms ensures the comparability and adaptability of results across different scenarios, thus providing a more comprehensive reference for dialogue generation and evaluation.

[0087] S70, the naturalness score, semantic consistency score, speaker consistency score, interaction rationality score and diversity score are fused to obtain the comprehensive quality score of the multi-speaker dialogue speech.

[0088] In this embodiment, the naturalness score, semantic consistency score, speaker consistency score, interaction rationality score, and diversity score of multi-speaker dialogue speech are multiple dimensions calculated independently, each reflecting different aspects of speech quality. However, a single dimension cannot comprehensively measure the overall quality. To obtain a unified comprehensive quality score, these dimensions need to be quantitatively integrated through a fusion mechanism.

[0089] First, the five scores need to be standardized to eliminate differences in the range of values ​​across different dimensions. For example, the naturalness score might be based on probability estimates from an acoustic model, while the semantic consistency score might be based on semantic embedding similarity; the two have different numerical scales. By using extreme value standardization or zero-mean standardization, the five scores can be uniformly mapped to the same range, forming a standardized quality feature vector.

[0090] Subsequently, the relative importance of each dimension in the comprehensive evaluation needs to be determined based on the actual application scenario. This process can be accomplished using a pre-set weight matrix. For example, in a virtual customer service scenario, semantic consistency and interaction rationality are more important than diversity; while in a film dubbing scenario, naturalness and diversity have greater weight. The weight allocation can be fixed or dynamically adjusted according to the activity level of the speakers in the dialogue, for example, increasing the weight of the dimension corresponding to the speaker who dominates the dialogue.

[0091] During the fusion process, standardized features and weight coefficients need to be weighted to obtain an initial comprehensive score. If time factors are considered, a temporal weighting strategy can be further introduced to give greater influence to the scores of more recent dialogue segments on the final result. The fusion calculation can be achieved through matrix multiplication, combining the weight matrix and feature vectors to generate a comprehensive score matrix, which is then aggregated to obtain a single score.

[0092] To enhance robustness, outlier removal is necessary for the initial composite score. Statistical methods, such as confidence interval estimation, can be used to exclude scores that exceed the confidence interval, avoiding distortion of the composite result due to local noise or anomalous behavior. Finally, the valid score set is aggregated and calculated, for example, using a geometric mean or weighted average, to generate the composite quality score.

[0093] In one implementation, the five scores are standardized to the [0,1] range through extreme value standardization, and the weights are set manually, which is suitable for use in scenarios where the evaluation rules are clear and the business is stable.

[0094] In another implementation, the weighting coefficients are learned automatically using machine learning methods. For example, training a model on medical consultation data automatically determines the importance of naturalness and interaction rationality; training a model on financial customer service data emphasizes semantic consistency and speaker consistency. This approach can automatically optimize weights based on large-scale corpora, improving adaptability.

[0095] A time-series weighting factor can also be introduced, giving a larger proportion of the score closer to the current moment in the overall result. For example, in real-time dialogue monitoring, the score of the most recent minute has a weight of 0.6, the previous stage has a weight of 0.3, and an even earlier stage has a weight of 0.1, thus ensuring that the overall result is more consistent with the real-time interaction status.

[0096] For outlier handling, the Monte Carlo method can be used to generate confidence intervals and eliminate scores with insufficient confidence; alternatively, robust statistical methods, such as median absolute deviation, can be used to reduce the interference of extreme values ​​on the final result.

[0097] This embodiment integrates naturalness score, semantic consistency score, speaker consistency score, interaction rationality score, and diversity score to form a unified comprehensive quality score, enabling a comprehensive evaluation of multi-speaker dialogue speech. Standardization ensures comparability across dimensions on the same scale, weight allocation ensures evaluation results better meet practical needs in different application scenarios, temporal weighting ensures the results dynamically reflect quality changes during the dialogue, and anomaly removal ensures the reliability of the final score. This fusion mechanism overcomes the limitations of single-index evaluation and improves the scientific rigor and applicability of speech generation and evaluation.

[0098] This invention relates to the field of speech processing technology and can be applied to business scenarios such as fintech and healthcare. It discloses a method, apparatus, device, and medium for analyzing multi-speaker dialogue speech, comprising: acquiring the multi-speaker dialogue speech to be analyzed; determining a naturalness score based on the acoustic features of the speech; determining a semantic consistency score based on the semantic embedding of the speech and the corresponding semantic embedding of the preset text; determining a speaker consistency score based on the embedding of multiple speakers of the same speaker; determining an interaction rationality score based on the duration of speech alternation and overlap; determining a diversity score based on the variance of the speech features; and fusing the naturalness score, semantic consistency score, speaker consistency score, interaction rationality score, and diversity score to obtain a comprehensive quality score for the multi-speaker dialogue speech. This invention quantifies multiple dimensions of indicators such as speech naturalness, semantic consistency, speaker consistency, interaction rationality, and diversity to construct a unified comprehensive quality scoring system. This allows the evaluation results to comprehensively reflect the naturalness and fluency, content matching, identity stability, interaction rhythm rationality, and feature richness of the multi-speaker dialogue speech. By fusing multiple dimensions of indicators, it avoids the one-sidedness caused by single-dimensional evaluation and achieves an objective, automated, and comparable comprehensive evaluation of the quality of multi-speaker dialogue speech.

[0099] In one embodiment, step S20 above includes:

[0100] S201, extract the acoustic features of the multi-speaker dialogue speech, standardize the acoustic features, and generate standardized acoustic features;

[0101] S202, Construct an acoustic feature vector containing temporal information based on the standardized acoustic features;

[0102] S203, input the acoustic feature vector into the pre-trained naturalness scoring model to obtain the corresponding naturalness score;

[0103] S204, the naturalness score is smoothed using a moving average or low-pass filter to generate the final naturalness score.

[0104] In this embodiment, the goal is to obtain a repeatable, comparable, and practical naturalness score based on the acoustic features of multi-speaker dialogue. The process begins with unified acoustic feature acquisition and standardization, followed by the construction of a representation carrying temporal information, which is then fed into a pre-trained naturalness scoring model to obtain the raw score. Finally, a smoothing strategy is used to suppress jitter and output a stable result. Each step is seamlessly connected using data format and time stamps to avoid intermediate state loss or ambiguity.

[0105] The acoustic feature extraction and standardization process first aligns the sampling rate and normalizes the amplitude of the multi-speaker dialogue speech. Typical settings include standardizing to 16 kHz sampling and linear amplitude normalization to the [-1,1] interval or target loudness. Pre-emphasis is then applied to enhance high-frequency information, followed by frame-by-frame windowing with a 25-millisecond frame width and a 10-millisecond step, using a Hamming window to reduce boundary leakage. Multiple sets of complementary features are calculated within each frame, covering short-time spectral envelope and fine-grained time-varying properties. Optional sets include log-Mel filter bank energy, Mel cepstral coefficients and first and second-order differences, fundamental frequency and confidence level, zero-crossing rate, short-time energy, spectral centroid and bandwidth, spectral roll-off point, spectral flux, jitter, and flicker. To reduce the impact of equipment differences and environmental noise, cepstral mean-variance normalization (CMVN), zero-mean unit variance normalization based on conversation or speaker statistics, or robust variants such as quantile scaling are used. When encountering scenarios involving endpoint silence and cross-speaker splicing, speech activity indicators and speaker segment identifiers are introduced to mask for missing speech frames and cross-segment statistical contamination, ensuring that standardized statistics are consistent with valid speech frames. The output of this stage is a standardized acoustic feature tensor aligned to a unified time axis, with a shape equal to the time step multiplied by the feature dimension, and includes a frame-level timestamp to support subsequent temporal operations.

[0106] The acoustic feature vectors containing temporal information are constructed based on the aforementioned tensors. To explicitly encode local dynamics, symmetric context windows centered on the current time step are concatenated, such as stacking features from several frames before and after each time step to form a context-enhanced vector representation. Simultaneously, differential and second-order differentials are preserved to characterize changes in velocity and acceleration, or statistics are aggregated at several scales through temporal pyramid pooling to express longer-range prosody. Boundary time steps are filled with mirror images or partially concatenated using effective frame masks to avoid temporal misalignment. If inference latency constraints are relaxed, the entire session can be used as a long sequence input; if online processing is required, a sliding window is used to generate overlapping blocks with accompanying causal masks. The output of this stage is a temporally aligned vector sequence that preserves frame-level details while explicitly carrying short-to-medium-range dynamics.

[0107] The pre-trained naturalness scoring model receives a sequence of temporal vectors and generates naturalness scores. The model can take the form of a joint structure of convolutional and gated recurrent or bidirectional attention, using one-dimensional convolution to extract local patterns, bidirectional long short-term units or self-attention to capture cross-syllable to phrase-level dependencies, and channel attention to emphasize key frequency bands and key time periods. The training objective is to regress to continuous scores within the interval [0,1] or ordered classification of discrete levels, with labels sourced from quality-controlled human intelligibility and fluency annotations or equivalent objective surrogate metrics. During the inference phase, raw score sequences are generated at the frame or segment level, while intra-segment convergence is performed using attention weights or parametric statistics to obtain the raw full-segment score. To ensure cross-dataset comparability, the model output is probabilistically calibrated, such as by isothermal regression or sigmoid scaling, and boundary clipping is applied to outlier activations. Silence frames and non-speech events are removed from loss calculation and inference output using prior masking to avoid structural bias in the scores.

[0108] Smoothing is applied to address temporal jitter and instantaneous spikes in the original score sequence. If frame-level scores are available, a fixed-length moving average filter is used, with the window length consistent with the duration from syllable to foot, such as three to seven frames, and boundary reflections are used to maintain the length. If higher-frequency jitter needs to be suppressed, a digital low-pass filter is designed, such as a second- or fourth-order Butterworth low-pass filter. The normalization cutoff frequency is selected based on the frame rate, often taking a value less than the upper bound of the syllable rhythm frequency. Zero-phase forward and backward filtering avoids phase delay. If the model directly outputs segment-level scores, an exponential moving average can also be performed on adjacent overlapping segments, with the attenuation coefficient depending on the real-time requirements of the application. The smoothed scores are then subjected to a final boundary pruning to ensure they fall within the range [0,1], preserving the mapping relationship with the original sequence for easy auditing and interpretable analysis.

[0109] The entire process is seamlessly integrated through the sharing of timestamps, masks, and standardized statistics. Frame-level timestamps ensure that features, scores, and filtering windows operate on the same timeline; masks are consistently effective during feature standardization, model inference, and smoothing stages; standardized statistics employ the same estimation strategy on both the training and evaluation sets to maintain score stability. Numerical stability measures protect against standardization with denominators close to zero, extremely short window operations, and initial filtering states, for example, by using minimum variance lower bounds, minimum effective frame counts, and filter steady-state warm-up to prevent abnormal outputs. The final natural score can be used as a single value or retain its time-series format for subsequent alignment and fusion with other dimensions.

[0110] This embodiment reduces the bias caused by speaker and recording environment differences through unified feature standardization. Temporal vector construction encodes prosody and coherence while preserving local details. A pre-trained scoring model outputs comparable scores across the regression space of different samples, and a smoothing strategy suppresses short-term spikes and random perturbations. These four interconnected steps ensure that the output reflects both microscopic fluency and clarity of pronunciation, as well as robustly presenting naturalness at the sentence group level, without distortion due to silence, device gain, or temporary noise. The resulting naturalness score maintains a consistent scale across different dialogues, speakers, and recording conditions, facilitating fusion evaluation with other quality dimensions and enabling stable operation in both offline and online scenarios.

[0111] In one embodiment, step S30 above includes:

[0112] S301, the speech embedding sequence of the multi-speaker dialogue speech is dynamically time-normalized and aligned with the semantic embedding sequence corresponding to the preset text to generate the aligned speech embedding sequence and the aligned semantic embedding sequence, and a temporal correspondence between the speech frame and the text unit is established.

[0113] S302, normalize the aligned speech embedding sequence and the aligned semantic embedding sequence respectively to generate normalized speech embedding sequence and normalized semantic embedding sequence;

[0114] S303, Based on the temporal correspondence, determine the frame-by-frame dot product value of the normalized speech embedding sequence and the normalized semantic embedding sequence, and generate the original similarity score sequence.

[0115] S304, Based on the voice activity detection results, filter the original similarity score sequence, remove invalid scores with confidence levels lower than a preset confidence threshold, and generate the remaining valid similarity score sequence;

[0116] S305, determine the dynamic weight coefficient matrix based on the speech energy intensity at the corresponding position of the remaining valid similarity score sequence;

[0117] S306, The remaining valid similarity score sequence is weighted and averaged using the dynamic weight coefficient matrix to generate the final semantic consistency score.

[0118] In this embodiment, the goal is to quantitatively measure the semantic matching degree between multi-speaker dialogue and preset text, forming a reproducible and comparable semantic consistency score. The input is a dialogue and its corresponding text, and the output is a single-valued or time-series score. The process sequentially completes cross-modal alignment, normalization, frame-by-frame similarity calculation, validity screening, energy-aware weighting, and numerical aggregation, all within the same timeline to avoid intermediate mismatches.

[0119] First, a speech embedding sequence and a semantic embedding sequence are constructed. The speech embedding sequence originates from segmenting the dialogue speech into equally spaced short time frames or frame-level or sub-word-level segments formed according to phoneme boundaries or word boundaries. These segments are then mapped to a fixed-dimensional vector space using a self-supervised or supervised acoustic encoder. The vectors are arranged in ascending order over time, retaining frame-level timestamps and confidence scores. The semantic embedding sequence originates from segmenting a predefined text into units that are alignable to the speech granularity, such as characters, words, or sub-words. These units are then mapped to equal-dimensional vectors using a semantic encoder, also carrying position indices on the text axis. To eliminate the risk of dimensional inconsistency during subsequent alignment, the two embedding sequences are aligned to the same dimension or dimensional mapping is completed through a linear mapping layer. Simultaneously, a mask is retained to identify invalid and padding units.

[0120] Dynamic time warping and alignment are then performed to establish cross-modal temporal correspondences. Using speech embeddings as the temporal axis and semantic embeddings as the reference sequence, an inter-frame cost matrix is ​​constructed. The cost is given by the distance metric between two unit vectors, typically one minus the normalized dot product or Mahalanobis distance. Under monotonic path and boundary constraints, the minimum cumulative cost path is calculated, outputting a one-to-one or one-to-many mapping from speech frames to text units, and providing the aligned speech embedding sequence and the aligned semantic embedding sequence. A window band constraint is introduced to suppress local path oscillations and limit the magnitude of deviation from the diagonal. To avoid misalignment of speech segments, silence rows and columns are masked in the cost matrix using speech activity masks. If there are significant changes in speech rate, additional step size penalties or slope constraints are introduced to ensure the stability of the path across fast and slow speech rate ranges. Finally, the temporal correspondence between speech frames and text units is obtained, serving as the unique index for subsequent frame-by-frame operations.

[0121] After establishing the alignment, the two aligned vectors are normalized. L2 norm normalization is used to project each vector onto a unit sphere, eliminating the impact of modulus differences on similarity. To avoid division-by-zero anomalies caused by zero vectors, a numerical threshold is set at the lower bound of the norm; vectors below the threshold are replaced with zero vectors and masked. After normalization, speech vectors and semantic vectors at the same time index are comparable, suitable for dot product similarity calculation.

[0122] The frame-by-frame dot product is calculated based on the temporal correspondence to form the original similarity score sequence. Since normalization has eliminated scale difference, the dot product is equivalent to cosine similarity, with a value range of [-1, 1]. To ensure that the results are in the non-negative interval for easy fusion, the scores can be linearly mapped to [0, 1]. When there are words with heavy accents or cross-domain texts, to avoid local extremely low values ​​affecting the overall result, a soft threshold mechanism is set to smoothly raise extremely low values ​​to an interpretable range, while recording the magnitude of the increase for subsequent confidence annotation.

[0123] The original similarity sequences are screened for validity using dual-channel signals: speech activity detection and model confidence. Speech activity detection identifies silent frames, noise frames, or non-speech sounds, while model confidence is derived from frame-level reliability during the embedding extraction or ASR alignment stages. The two signals are then ANDed to form a valid frame mask. Scores below a preset confidence threshold are discarded, and the remaining valid similarity score sequences proceed to the next stage. To avoid insufficient sample usability due to continuous large-scale rejection, a minimum effective frame ratio is configured. When the threshold is lowered, the process either reverts to single-channel screening based solely on confidence or lowers the threshold for a one-time relaxation.

[0124] Speech energy intensity is introduced into the effective frames to establish a dynamic weighting coefficient matrix. Energy intensity is calculated from the original speech frames using short-time energy or logarithmic power, and then normalized within the dialogue to obtain a comparable scale. Weights can be determined solely by energy or combined with confidence scores to form joint weights. Common forms include the product or weighted sum of energy and confidence scores, which are then normalized to the probability distribution within each time window using Softmax or maximum value normalization to ensure numerical stability and interpretability. The time dimension of the weighting matrix is ​​aligned with the effective score sequence, and the vector dimension is one-dimensional or expanded by sub-speaker and sub-channel to adapt to different aggregation granularities.

[0125] Finally, a weighted average is calculated under the combined constraints of the effective scores and weights to obtain the final semantic consistency score. In linear aggregation, the output is a weighted sum divided by the sum of weights. In anomaly-sensitive scenarios, a weighted robust mean with Huber loss can be used to limit the impact of extreme values ​​to a controllable range. In scenarios where it is desirable to highlight highly consistent segments, power-law weighting can be introduced to soft-focus high-scoring regions. If both intra-segment and full-segment results are required, a sliding window-level time series score and a full-segment scalar score are output simultaneously, along with quality annotations such as the effective frame ratio and weight entropy, facilitating subsequent fusion and auditing. The entire process shares timestamps, masks, and scales throughout, and all intermediate quantities can be traced back to the original frames, ensuring interpretability and testability.

[0126] This embodiment establishes a stable mapping between speech temporal sequence and text units through cross-modal alignment, eliminates embedding modulus differences through normalization, and uses dot product metrics to directly reflect semantic proximity on a unit sphere. Invalid and unreliable frames are eliminated based on speech activity and confidence levels. Energy-aware weights highlight pronunciation segments with more complete semantic content. The final weighted aggregation suppresses noise interference while retaining sensitivity to true matching relationships. The semantic consistency score obtained through this path remains scale-stable under different speakers, speech rates, and channel conditions, and can be seamlessly aligned and integrated with subsequent dimensions. It supports automated evaluation and online monitoring of dialogue generation quality with lower variance and higher discriminative power.

[0127] In one embodiment, step S40 above includes:

[0128] S401, group the speech segments in the multi-speaker dialogue speech based on the speaker identity identifier, extract speaker embeddings for at least two speech segments of the same speaker, and sort the speaker embeddings in ascending order according to the start timestamp of the speech segments to construct a speaker embedding time series;

[0129] S402, determine the cosine similarity of speaker embedding vectors at adjacent positions in the speaker embedding time series, and generate a time series similarity matrix;

[0130] S403, Based on the temporal similarity matrix, determine the stability index of speech feature changes across positions for the same speaker;

[0131] S404, detect abrupt changes in the speech embedding vector in the speaker's embedded time series by using a sliding window, and determine the total number of valid switching events by combining the temporal similarity threshold;

[0132] S405, determine the switching event frequency based on the total number of valid switching events and the total number of interactive units in the multi-speaker dialogue voice;

[0133] S406, Based on the stability index and the switching event frequency, generate the speaker consistency score of the multi-speaker dialogue speech.

[0134] In this embodiment, the goal is to measure the stability and identity continuity of the same speaker's voiceprint in multi-speaker dialogues using a unified scale, thereby generating a traceable speaker consistency score. The input consists of the segmentation results of the dialogue speech and the speaker identifier for each segment. The output is a consistency measure in scalar or time-series form, which may include quality annotations for subsequent fusion.

[0135] First, speech segments are grouped based on speaker identifiers. Speaker identifiers are derived from voiceprint clustering tags, registration database matching numbers, or steady-state tags output by real-time recognition models during online or offline separation processes, requiring uniqueness and temporal continuity within the same session. Speech segments are speech intervals obtained through endpoint detection, acoustic decoding alignment, or segmentation models. Each interval carries a start timestamp, end timestamp, and confidence level. For each unique speaker identifier, at least two speech segments are collected, arranged in ascending order of start timestamps, resulting in a strictly increasing time index, constructing the skeleton of the speaker's embedded time series.

[0136] Speaker embeddings are then extracted for each selected segment. Speaker embeddings map the time-domain or frequency-domain speech representation to a vector representation in a fixed-dimensional space, derived from an end-to-end speaker network or statistical backend, maintaining consistent dimensionality for comparison within the same space. To reduce channel interference and length variations, segments are aligned and normalized in the feature domain before being fed into the embedding extractor; if a segment is too short, adjacent segments are concatenated or data augmentation is used to achieve the minimum effective duration. After extraction, the embedding vectors are arranged in ascending order of the speech segment's start timestamp, forming an ordered speaker embedding time series. The timestamp, segment confidence, and effective duration of each vector are retained for subsequent weighting and filtering.

[0137] The cosine similarity of adjacent vectors is calculated on the time series to form a temporal similarity matrix. Adjacent positions refer to two segments that appear consecutively on the time axis. This can be achieved by using first-order adjacency to form the main diagonal band, or by extending to multiple-order adjacency within a fixed window to improve robustness. Vectors are first L2 normalized, and the cosine similarity is equivalent to a dot product, with values ​​ranging from -1 to 1. To facilitate subsequent aggregation, linear mapping or temperature scaling is used to map the similarity to zero to one. The similarity of all adjacent pairs is filled into a banded matrix. The main band reflects short-term continuity, while the secondary band can be used for mid-term drift observations. Low-confidence segment pairs are masked to prevent noise from dominating the statistics.

[0138] A stability index is calculated based on a temporal similarity matrix to quantify the stability of a speaker's characteristics over time. The stability index comprises two dimensions: central tendency and dispersion. Common forms include the ratio of the weighted mean to the weighted standard deviation, or one minus the weighted coefficient of variation. The weights are derived from a joint function of the effective duration and confidence level of a segment; longer segments and higher-confidence segments contribute more, reducing the influence of occasional short segments on the statistics. To avoid extreme value bias, robust statistics can be introduced, such as quantile truncation or Huber suppression on the similarity sequence before calculating central tendency and dispersion. The stability index monotonically increases with stability in the range of zero to one, while retaining the weighted entropy and the effective sample ratio as quality indicators.

[0139] A sliding window abrupt change detection method is used on time series data to identify abrupt changes in speech embedding vectors. The sliding window is defined as a local range of fixed duration or a fixed number of segments. Within the window, short-term baselines and deviation indices of adjacent similarities are calculated, and the location of sudden drops is located using statistical significance tests or threshold strategies. To improve accuracy, a temporal similarity threshold is introduced as a secondary criterion; abrupt changes are marked only when the local similarity is below the threshold and the decrease exceeds a set boundary. Multiple adjacent markers are morphologically merged to avoid duplicate counting of the same event, resulting in a total number of effective switching events. This counting is performed independently on a per-speaker basis, and identity differentiation is maintained during session-level merging to avoid cross-identity interference.

[0140] The switching event frequency is calculated based on the total number of valid switching events and the total number of interaction units. The total number of interaction units is defined as the time metric used for normalization in a session, which can be the number of dialogue rounds, the number of speaking turns, or the number of equal-length time windows. The selected metric must be comparable across different sessions and consistent with the upstream segmentation strategy. The switching event frequency is the ratio of the number of events to the number of metric units, reflecting the intensity of identity mutation per unit interaction density in the range of zero to one. To avoid estimation bias caused by sample sparsity, Laplace smoothing or Bayesian calibration is introduced, and a small constant is added to the denominator or a prior distribution is used for posterior estimation of the frequency.

[0141] Finally, the stability index and the frequency of switching events are jointly mapped to a speaker consistency score. The mapping function follows monotonicity and interpretability; a positive influence from the stability index leads to an increase in the consistency score, while a negative influence from the switching event frequency leads to a decrease in the consistency score. In linear mapping, one can be used to subtract the weighted frequency multiplied by the stability factor, or a fusion of logarithmic and power exponent domains can be used to enhance the separation in extreme cases. To improve robustness, temporal smoothing is added to the output, and the effective sample ratio and the mean event confidence score are retained as audit indicators. When a session contains multiple clustering branches with the same identity, minimum cut consistency correction is used to correct cross-segment drift, avoiding misjudgment of cluster drift as identity switching. A unified timestamp and identity index are shared throughout the process, ensuring that each value can be traced back to a specific segment and time.

[0142] This embodiment transforms speaker representations into measurable stability and switching intensity through a modeling approach that aggregates by identity, orders by time, and compares adjacent segments. The stability index uses weighted centralization and discrete quantities to characterize consistency within the same identity over time, while the switching event frequency uses normalized counts to characterize identity coherence breaks. Both form a unified consistency score under monotonic mapping. Compared to approaches relying solely on global mean or a single threshold, this structure maintains discriminative power even with noisy segments, short-term anomalies, and cluster drift. Furthermore, it amplifies reliable information and suppresses unreliable segments through weights and masks, thereby outputting a stable scale under different session lengths, segmentation granularities, and recording conditions. This facilitates integration with other dimensions into a comprehensive quality index.

[0143] In one embodiment, step S50 above includes:

[0144] S501, Detect overlapping regions of speech streams in the multi-speaker dialogue, and record the start and end timestamps of each overlapping region;

[0145] S502, determine the actual speech overlap duration of the overlapping area based on the start timestamp and the end timestamp;

[0146] S503, extract the theoretical duration of alternating overlapping speech in the preset reference dialogue text;

[0147] S504, Based on the time difference between the actual speech overlap duration and the theoretical duration, determine the time synchronization error rate;

[0148] S505, Based on the time synchronization error rate and the preset interaction rhythm threshold, generate an interaction rationality score.

[0149] In this embodiment, the goal is to measure, based on a timeline, whether alternations and overlaps in multi-speaker dialogues conform to the expected rhythm, generating a quantifiable interaction rationality score. Inputs include timestamps of the multi-speaker dialogue speech that can be aligned with audio trajectories and corresponding speech activity markers or separated speaker trajectories, as well as reference dialogue text marked with alternation and allowed overlap indicators. Outputs are scores ranging from zero to one and traceable intermediate statistics.

[0150] First, overlapping region detection is performed. The dialogue audio is discretized under a fixed frame length and frame shift to generate a frame-by-frame speech activity mask. The speech activity source can be a detector jointly determined by energy and spectral entropy, or a frame-by-frame speech probability inferred by a deep model, which is converted into a binary activity signal through thresholding and hysteresis rules. If speaker separation or segmentation results are available, an independent activity trajectory is generated for each speaker, and the trajectory is timestamped. If necessary, sampling rate rounding errors are eliminated through interpolation or integer sampling point alignment. At the frame level, the activity signals of each speaker are subjected to parallel logical AND operations. If the activity of at least two speakers in the same frame is true, it is marked as an overlapping frame. Consecutive overlapping frames are merged into intervals to obtain a sequence of start and end timestamps. To reduce jitter false detections, a minimum duration and minimum interval merging strategy is introduced to suppress isolated intervals shorter than the threshold and merge adjacent intervals with an interval less than the threshold. The obtained timestamps are recorded with a unified clock and carry the minimum or weighted average confidence of the source trajectory as the confidence of the interval.

[0151] Next, the actual speech overlap duration is calculated. Global overlap is defined as the total time spent simultaneously speaking on the union of all speaker activity trajectories. The total global overlap duration is obtained by subtracting the start timestamp from the end timestamp of each overlap interval and summing the results. To differentiate the details of dialogue alternation quality, pairwise overlap durations can be calculated for each pair of speakers, and then the global metric can be obtained using the maximum, average, or weighted sum. To reduce the impact of low-confidence intervals on duration, interval confidence weighting is introduced. The interval lengths are linearly or exponentially weighted according to interval confidence before aggregation. For scenarios with slight start and end boundary offsets, fixed millisecond-level boundary contraction can be performed at both ends of each interval to avoid artificially inflated durations caused by endpoint spikes.

[0152] The theoretical overlap duration is then extracted from the reference dialogue text. The reference text should clearly define the expected boundaries and allowed overlap windows for each speaker rotation. For example, the annotation file can define speaking intervals using role labels and time markers, or use a marker syntax to specify the time ranges for interruptions, agreement, and interjections. The theoretical set of overlap intervals is obtained through parsing and then aggregated using a standard consistent with the actual duration. If the reference text is plain text without time markers, a mapping between text and audio can be established through forced alignment or automatic transcription and time stamping, thus projecting the alternation markers in the text onto the timeline to form theoretical intervals. To improve robustness, the theoretical intervals can be merged using the same minimum duration and minimum interval merging process, ensuring consistency in the calculation standards for both types of durations.

[0153] After obtaining both actual and theoretical durations, the time synchronization error rate is calculated. The error can be represented by either a signed or absolute difference. To ensure comparability between different sessions, a normalized denominator is introduced. Normalization can use the total dialogue duration, the theoretical total overlap duration, or the average of the actual and theoretical durations. A common form is that the error rate equals the actual total overlap duration minus the theoretical total overlap duration, then divided by the total dialogue duration. To avoid the amplification effect caused by extremely short sessions, a lower bound protection is introduced by adding a very small positive number to the denominator or using the lower bound of the total dialogue duration for truncation. Considering the potential systematic time offset between the reference text and audio, a global time offset estimation can be performed before calculating the error rate. This involves searching within a small offset range to minimize the offset of the overlap difference between the two, and then comparing the durations using the corrected intervals, thereby reducing the systematic impact of alignment errors.

[0154] Finally, an interaction rationality score is generated based on the time synchronization error rate and the interaction rhythm threshold. The threshold represents the acceptable overlap deviation range in the target scenario, derived from historical data statistics or business specifications. The scoring function needs to satisfy monotonicity and boundedness. A common practice is to assign a full score when the error rate does not exceed the threshold, and reduce the score linearly or exponentially when it exceeds the threshold. To suppress the impact of occasional anomalies, a sliding window smoothing can be applied to the error rate time series before inputting it into the scoring function; alternatively, a moving average or low-pass filtering can be applied to the scoring results to output a stable final result. If both insufficient and excessive overlap are considered unreasonable, a piecewise function with absolute value or symmetric penalty can be used on the error rate to make the score peak near the theoretical value and then decrease to both sides. Throughout the process, intermediate products such as the overlap interval list, confidence weight, theoretical interval list, and global bias are retained for auditing and parameter tuning.

[0155] This embodiment uses frame-level overlap detection and timestamp interval merging to form a traceable actual overlap duration, text alignment to obtain a theoretical overlap duration consistent with this, normalized error rate to eliminate the impact of session duration differences, and finally thresholding mapping to obtain an interaction rationality score. This link transforms the rhythmic attributes of dialogue alternation and overlap into a unified-scale quantitative indicator, which can both penalize the congestion caused by excessive overlap and identify the communication delay caused by insufficient overlap. By using confidence and minimum persistence strategies to reduce the interference of noise segments and boundary jitter, and by using global time offset correction to reduce the systemic impact of text and audio alignment errors, it maintains stable and consistent evaluation results under different scenarios, session durations, and acquisition conditions, facilitating subsequent integration with other dimensions into a comprehensive quality score.

[0156] In one embodiment, step S60 above includes:

[0157] S601, Extract the voiceprint feature vectors of each speaker in the multi-speaker dialogue speech, and construct a multi-dimensional speech feature sequence;

[0158] S602, determine the variance value of each dimension of the feature in the multidimensional speech feature sequence, and generate a feature variance vector;

[0159] S603, normalize the feature variance vector to generate a normalized variance vector;

[0160] S604, The normalized variance vector is weighted and fused according to the preset weight coefficients to generate a comprehensive variance value;

[0161] S605, Based on the comprehensive variance value and the preset diversity threshold range, generate the diversity score of the multi-speaker dialogue speech.

[0162] In this embodiment, multi-speaker dialogue speech is used as input. Speech activity marking and speaker attribution need to be completed on the timeline first to ensure that subsequent measurements can be performed at the speaker level. Speech activity can be determined through joint decision-making using energy and spectral features or by generating binary trajectories using a probability-based detection model; speaker attribution can be achieved by segmentation and clustering to obtain continuous speech segments and corresponding identity labels. Non-speech segments such as silence, music, and ambient noise are removed from the timeline to avoid shifting the statistics. The resulting dialogue timeline is divided into several speech segments, each segment being bound to a unique speaker identity and start and end timestamps.

[0163] Voiceprint feature vectors are used to characterize the differences in speech at the short-term and speaker steady-state scales. Short-term components may include Mel-frequency cepstral coefficients, fundamental frequency, bandpass energy, harmonic-to-noise ratio, formant trajectories, zero-crossing rate, and modulation spectrum amplitude; steady-state components may include speaker embedding, channel robust embedding, and timbre-related statistics. To ensure comparability, the time axis is discretized with a fixed frame length and frame shift. For each frame, the aforementioned dimensions are extracted and aggregated according to the speaker's identity. Under each speaker's name, all spoken frames for that speaker are sorted chronologically and concatenated to form a multi-dimensional speech feature sequence. To reduce the influence of scene noise, confidence weights are introduced for outlier frames during the construction process; low-confidence frames are included with lower weights in subsequent statistics.

[0164] Variance values ​​characterize the dispersion of features across the global dialogue span. Statistical variance is calculated dimension-wise for the multidimensional speech feature sequence, employing robust variances insensitive to outliers when necessary, such as estimates based on median absolute deviation or second-moment estimates with Huber loss. To avoid bias caused by differences in speaker speaking proportions, weighted variances are applied based on speaking duration or frame count, ensuring each speaker's contribution is proportional to their effective speech volume. To suppress the amplification effect of extremely short segments on statistics, intra-segment variance can be calculated first at the segment level, then combined with the global level at the speaker level using segment duration as weight. The above dimension-wise results are concatenated into a feature variance vector, retaining the corresponding dimension names and calculation methods to ensure traceability.

[0165] Normalized variance vectors are used to eliminate differences in dimensions and inconsistencies in numerical scales. Extreme value standardization can be used to linearly map each dimension of variance to the zero-to-one range within the historical distribution or business calibration interval; L2 normalization can be used to normalize the entire feature variance vector according to the vector norm, emphasizing relative structure while weakening absolute magnitude; quantile scaling can also be used to stretch and compress the variance according to the upper and lower quantiles of the distribution, improving stability against long-tailed noise. The normalization strategy is fitted offline on the full dataset or a subset of scenarios, and then solidified into an online parameter table to ensure comparability across different sessions.

[0166] The preset weight coefficients are used to express the differences in the contribution of different dimensions to diversity. The weights can be derived from empirical calibration, optimization on a validation set, or learning through task objectives. To improve generalization ability, multiple sets of weights can be configured according to scenario type, collection conditions, or target business needs. For example, in scenarios emphasizing emotional richness, the weights of fundamental frequency correlation and energy correlation dimensions can be appropriately increased; in scenarios emphasizing timbre differences, the weights of speaker embedding statistics can be increased. The weight coefficients and the normalized variance vector are multiplied element-wise and summed to obtain the comprehensive variance value. Alternatively, a Softmax algorithm with a temperature parameter can be used to constrain the weights to a probability distribution to reduce the exclusive influence of a single dimension. If it is necessary to incorporate structural information such as the number of speakers and rotation frequency, an adjustment factor can be added to the calculation of the comprehensive variance value to make the indicator respond differently to "many speakers with complementary characteristics" and "many speakers with homogeneity."

[0167] The diversity threshold range is used to map the comprehensive variance value to a diversity score with a uniform scale. The threshold range is obtained through historical data evaluation or statistical analysis of labeled samples. The lower limit represents the insufficient diversity area, which may correspond to simple speech features, flat intonation, and monotonous rhythm; the upper limit represents the excessively discrete area, which may correspond to excessive emotion, unstable vocalization, and noise disturbance. When the comprehensive variance value falls within the range, it is mapped to a high score using a linear or slowly varying function; below the lower limit, it decreases according to a convex function with the difference, penalizing insufficient diversity; above the upper limit, it decreases according to a concave function to avoid excessively penalizing occasional high-energy bursts. To improve stability, the comprehensive variance value can be time-smoothed before interval mapping; conversation-level regularization can also be applied to the diversity score to keep the scoring variance of long and short conversations within a controllable range. The final output diversity score is saved along with intermediate values ​​used for auditing, supporting source tracing and parameter adjustment.

[0168] This embodiment constructs a multi-dimensional speech feature sequence at the speaker level and obtains the feature variance vector through robust statistics. Then, it performs unified scale normalization and weight fusion to obtain a comprehensive variance value. Finally, it completes interpretable mapping within a diversity threshold range to form a diversity score for multi-speaker dialogue speech. This process quantifies the discreteness of the temporal, spectral, and embedding domains using a unified standard, distinguishing between situations where "features are too concentrated, leading to monotony" and "excessive feature dispersion may originate from noise or distortion." Configurable emphasis is applied to different dimensions through weight coefficients, allowing the scoring to adapt to different business objectives. Robust variance and quantile normalization suppress the influence of abnormal frames, while temporal smoothing and interval mapping improve score stability and comparability. This provides reliable input for subsequent fusion with dimensions such as naturalness, semantic consistency, speaker consistency, and interaction rationality, improving the accuracy and robustness of the comprehensive quality assessment.

[0169] In one embodiment, step S70 above includes:

[0170] S701, the naturalness score, semantic consistency score, speaker consistency score, interaction rationality score and diversity score are respectively subjected to extreme value standardization processing to generate a standardized quality feature vector;

[0171] S702, based on the preset scene weight matrix and the proportion of active time of each speaker in the dialogue, determines the dynamic weight coefficient vector of naturalness score, semantic consistency score, speaker consistency score, interaction rationality score and diversity score through weighted processing.

[0172] S703, based on the temporal progress of the dialogue, apply an exponentially decaying weight to the standardized quality feature vector to generate a time-weighted feature matrix that dynamically changes over time.

[0173] S704, the dynamic weight coefficient vector and the time-series weighted feature matrix are weighted by dimension to obtain an initial value of the comprehensive quality score containing spatiotemporal features;

[0174] S705, by estimating the confidence interval of Monte Carlo, outlier data points with confidence levels lower than a preset threshold are removed from the initial value of the comprehensive quality score to obtain the remaining valid score;

[0175] S706, Perform geometric averaging on the remaining valid scores to generate a comprehensive quality score for the multi-speaker dialogue speech.

[0176] In this embodiment, the naturalness score, semantic consistency score, speaker consistency score, interaction rationality score, and diversity score are five independent quality indicators extracted from multi-speaker dialogue speech. They differ in numerical range, units, and numerical distribution. Direct weighting or fusion might lead to some dimensions dominating the results due to their large value ranges. Therefore, extreme value standardization is necessary. Extreme value standardization maps each score to a zero-to-one range based on its historical minimum and maximum values. This ensures that the five dimensions are compared on the same numerical scale. Standardization yields a standardized quality feature vector, where each dimension corresponds to a standardized quality score, providing a unified input for subsequent weight allocation and temporal modeling.

[0177] After obtaining the standardized quality feature vector, weights need to be assigned to different scenarios and speakers. The scenario weight matrix expresses the importance ratio of each dimension in a specific use case. For example, semantic consistency might have a higher weight in a courtroom scenario, while interaction rationality is more critical in a customer service scenario. Simultaneously, the speaker's active duration ratio in the dialogue measures the contribution of different speakers to the overall dialogue; speakers with longer speaking times should have a higher weight in the overall fusion. By combining the scenario weight matrix with the active duration ratio, a dynamic weight coefficient vector can be generated. This vector assigns specific weights to each dimension, enabling the fusion process to adapt to different dialogue scenarios and speaker structures.

[0178] The standardized feature vectors are not only scene-dependent but also exhibit temporal variability. Performance at different points in a dialogue may carry different weights, with recent statements often being more indicative of overall quality than earlier ones. Therefore, an exponentially decaying weight is required. Specifically, the time series of the standardized quality feature vectors is weighted according to their temporal differences, with scores closer to the current moment receiving higher weights and scores further back receiving lower weights. The resulting temporally weighted feature matrix preserves the dynamic characteristics of score changes over time, more accurately reflecting the dialogue's quality performance over time.

[0179] Combining the dynamic weight coefficient vector with the temporal weighted feature matrix requires fusion through dimensional weighting. Dimensional weighting means weighting the dynamic weight coefficient of each dimension with its value in the temporal weighted matrix to obtain an initial value for the fused overall quality score. This initial value simultaneously includes the static weights of the scene dimension and the dynamic decay effect of the time dimension, thus reflecting the distribution of the quality score in both spatiotemporal dimensions.

[0180] Since actual speech data may contain noise, abnormal speech, or computational errors, outlier removal is necessary after fusion. Monte Carlo confidence interval estimation is a statistical method that assesses the distribution range of initial scores by constructing a large number of sampled simulations and determines confidence intervals based on a set confidence level. Scores falling outside the confidence interval are identified as outliers and removed, while the remaining scores form the valid score set. This avoids distortion of the overall score due to isolated extreme values.

[0181] After obtaining the remaining valid scores, a final comprehensive quality score needs to be generated. Geometric averaging is a common method for fusing numerical values ​​across multiple dimensions. It effectively suppresses the extreme influence of excessively high or low values ​​in certain dimensions on the overall result. By calculating the geometric mean of the remaining valid scores, the result is smoother and more stable, while maintaining a relative balance among the multiple dimensions. The final comprehensive quality score can serve as a quantitative result of the overall quality of multi-speaker dialogue speech, used to evaluate the quality of different speech samples.

[0182] For example, naturalness fractions:

[0183] The naturalness score N is calculated using acoustic features such as Mel spectrogram, pitch F0, and intensity I.

[0184] N=α1·MAE(Mel gen Mel ref )+α2·Var(F0)+α3·Var(I)

[0185] Where α1, α2, α3 are weighting coefficients, Mel gen To generate Mel spectrograms for speech, Mel ref Mel spectrogram of the reference speech.

[0186] Semantic consistency assessment:

[0187] Using semantic embeddings text With voice embeddings audio Cosine similarity calculation:

[0188]

[0189] Speaker consistency assessment:

[0190] Using speaker embedding v i Assessing consistency of the same speaker across multiple turns of conversation:

[0191]

[0192] The basic idea is to calculate the cosine similarity between the speaker's embedding vectors for each two speeches in a multi-turn dialogue, and take the average value as the speaker's consistency score in the whole dialogue.

[0193] Where K represents the number of turns the speaker makes. For example, if a speaker makes 5 turns in a dialogue, then K = 5. i ,v jThis represents the speaker embedding vector extracted from different speech segments. A speaker embedding vector is a high-dimensional vector representation used to capture a speaker's vocal features, such as timbre, vocal tract characteristics, and pronunciation habits. The vector representations of different speech segments may vary due to the speech environment, speech rate, or background noise. This represents the normalization factor, used to calculate the average. Since there are a total of K(K-1) ordered pairs (i.e., different combinations of segments), it is used to divide the total number to obtain the overall uniform mean.

[0194] Interaction rationality assessment:

[0195] Define dialogue alternation and overlap scoring I:

[0196]

[0197] in, This refers to the overlap duration in the generated speech. Here, overlap duration refers to the total time during which speech segments from different speakers overlap in an automatically generated multi-speaker dialogue. t represents the overlap duration in the reference speech. The reference speech here is standard speech dialogue data (human or real dialogue corpus) used for comparison. This parameter reflects the actual alternation and overlap of speech in the reference speech. total This represents the total dialogue duration. It refers to the overall duration from the start to the end of the dialogue, used for normalization to ensure the comparability of metrics across different dialogue durations. I represents the dialogue alternation and overlap score. The time synchronization error is calculated by comparing the difference between the overlap duration of the generated speech and the overlap duration of the reference speech, and then normalized based on the total duration to obtain the final score. Its value typically ranges from 0 to 1; the closer the value is to 1, the closer the generated speech and reference speech are in their alternation and overlap patterns, and the higher the reasonableness of the interaction.

[0198] Diversity assessment:

[0199] The diversity score D is calculated by analyzing the variance of the timbre, speech rate, and emotional features of the generated speech.

[0200] D = Var(f timbre )+Var(f rate )+Var(f emotion )

[0201] D is the final diversity score. It measures overall diversity through the variance of multiple feature dimensions of the speech. timbre The timbre characteristics of speech are typically described using features such as Mel-frequency cepstral coefficients (MFCC) and spectral envelope. The more dispersed the distribution of different timbres, the more diverse the timbre representation of the generated speech. rateSpeech rate, or the speed of pronunciation per unit of time, can be measured by speech frame length, word duration, or pronunciation intervals. Variations in speech rate reflect changes in the speaker's rhythm in different contexts, increasing the naturalness of the conversation. emotion Emotional features can be represented, such as those obtained through acoustic features (fundamental frequency F0, energy, pitch curve) or emotion vectors output by emotion classification models. A larger variance in the emotional feature indicates a wider range of emotional expression covered by the generated speech, thus increasing diversity. `Var()` represents the variance operator, used to measure the dispersion of this feature across all speech segments. A larger value indicates richer variability in that dimension.

[0202] Overall rating:

[0203] The five dimensions are weighted and combined to form the final score Q:

[0204] Q = w N N+w S S+w C C+W I I+w D D

[0205] Among them, w N w S w C w I w D These are the weighting coefficients for each dimension, which can be adaptively adjusted according to the actual application scenario.

[0206] Example: In remote rehabilitation guidance scenarios within the healthcare field, patients typically need to engage in multiple rounds of voice interaction with health management personnel or intelligent health assistants to obtain personalized rehabilitation training suggestions and lifestyle adjustment recommendations. In this scenario, accurately assessing the naturalness of the voice interaction, the accuracy of semantic communication, the consistency of the speaker's identity, and the rationality and diversity of multi-party communication directly impacts whether the patient can clearly understand the instructions and actively cooperate with the rehabilitation process.

[0207] First, it is necessary to acquire the multi-speaker dialogue audio to be analyzed. This audio may come from recordings of interactions between patients, rehabilitation instructors, and family members. By acquiring the raw audio stream and converting its format, consistency in subsequent feature extraction can be ensured. Next, acoustic features, such as Mel-frequency cepstral coefficients (MFCC), fundamental frequency, and formant parameters, are extracted from the audio signal. These features are standardized and constructed into a time-series vector, which is then input into a trained naturalness scoring model. This model outputs a speech naturalness score, which measures whether the patient's or instructor's speech is fluent and whether there are excessive pauses or unnatural synthetic traces during the interaction.

[0208] Following naturalness analysis, semantic consistency needs to be evaluated. By embedding and aligning the speech with reference rehabilitation guidance text or pre-defined text embeddings in a knowledge base using dynamic temporal warping, the temporal correspondence between speech and text can be obtained. Further, frame-by-frame dot product similarity is calculated, and low-confidence frames are removed using speech activity detection results, resulting in a valid set of similarity scores. A dynamic weighted coefficient matrix is ​​constructed by combining the speech energy intensity distribution, and then a weighted average is performed to generate a semantic consistency score. This quantifies the consistency between the spoken communication and the medical instruction text, ensuring that patients accurately receive the rehabilitation guidance content.

[0209] In multi-person dialogues, speaker consistency also needs to be verified. By grouping speech segments based on identity identifiers, multiple embedding features of the same speaker are extracted and a time series is constructed. The cosine similarity of adjacent speech embedding vectors is calculated to obtain a temporal similarity matrix, which is used to assess the stability of speech feature changes. Combining sliding window detection of abrupt changes with a threshold to determine the frequency of switching events, a speaker consistency score can be generated. This score reflects the continuity of the patient's or healthcare worker's identity in the dialogue, avoiding communication confusion caused by speaker identification errors or frequent switching.

[0210] The rationality of dialogue interactions also needs to be quantified. By detecting overlapping areas in the speech stream and recording start and end timestamps, the actual duration of speech overlap is obtained. This is compared with the theoretical duration in the reference rehabilitation conversation text to calculate the time synchronization error rate, and then combined with a preset interaction rhythm threshold to generate an interaction rationality score. This score can measure whether the alternation of speech between the patient and the instructor is reasonable, avoiding excessive speech overlap or prolonged interruptions, thereby ensuring the efficiency and comprehensibility of rehabilitation guidance.

[0211] Finally, the diversity of speech is analyzed. By extracting the voiceprint feature vectors of each speaker and constructing a multidimensional sequence, the variance of each dimension of the feature can be calculated, generating a variance vector. This variance vector is normalized and weighted to obtain a comprehensive variance value, which is then compared with a diversity threshold range to generate a diversity score. This score is used to measure whether the speech performance of patients and rehabilitation personnel is rich enough, such as whether it has intonation variations, rhythm differences, and expressive diversity, thereby improving the natural experience of interaction.

[0212] After calculating the above five scores, they need to be integrated. Extreme value standardization is used to unify the scores from different dimensions to the same scale. A dynamic weight vector is generated by combining the scene weight matrix with the speaker's active duration ratio. An exponentially decaying weight is applied to the score vector to generate a time-weighted feature matrix. The dynamic weights and time-series features are then weighted dimensionally to obtain an initial comprehensive quality value. After removing outliers using Monte Carlo confidence interval estimation, a geometric mean is calculated on the valid scores to generate the comprehensive quality score. The final comprehensive score comprehensively reflects the overall performance of voice interaction in terms of naturalness, accuracy, rationality, and diversity, providing quantitative feedback for the rehabilitation training system.

[0213] In the healthcare field, it can be applied to remote rehabilitation voice guidance, helping the platform to automatically identify whether patients correctly understand and perform rehabilitation actions, and to quantitatively monitor the quality of instructors' voice feedback, thereby improving the standardization and personalization of rehabilitation services.

[0214] In remote customer service and intelligent consultation scenarios within the fintech sector, financial institutions need to handle numerous multi-round voice interactions between customers and intelligent or human customer service representatives. These interactions encompass not only routine business inquiries but also high-value aspects such as risk warnings, contract explanations, and compliance confirmations. Therefore, the quality of these voice interactions directly impacts customer experience, business compliance, and the efficiency of financial services. To ensure high-quality and controllable interactions, a comprehensive analytics process can be used to conduct multi-dimensional quality assessments of customer-customer dialogues.

[0215] First, it is necessary to acquire the multi-speaker dialogue voice recordings to be analyzed. These recordings may come from customer calls to voice hotlines or conversation recordings of intelligent voice customer service systems. The acquired voice signals undergo standardized formatting to ensure the stability and consistency of subsequent feature extraction. Next, acoustic features, including fundamental frequency, energy distribution, formants, and other parameters, are extracted from the dialogue voice. These features are then standardized to construct a temporal feature vector. This vector is input into a pre-trained naturalness scoring model, which outputs a naturalness score to measure whether the customer service or customer's voice is clear and fluent, and whether there are any unnatural phenomena caused by synthesis or excessive modification.

[0216] After the naturalness assessment is completed, a semantic consistency score needs to be calculated. By extracting the customer's speech embedding and dynamically time-warping it to align with the semantic embedding of pre-defined business text (such as contract terms and risk warnings), a frame-by-frame correspondence between speech and text can be established. Then, by calculating frame-by-frame dot product similarity and removing low-confidence segments based on speech activity detection results, a sequence of valid similarity scores is generated. Furthermore, a dynamic weight matrix is ​​generated using speech energy distribution, and the valid scores are weighted and averaged to obtain the semantic consistency score. This score measures whether the customer accurately restated the compliance confirmation content or whether the customer service representative faithfully conveyed the contract terms.

[0217] In interactions involving multiple speakers (such as customers, customer service representatives, and compliance specialists), speaker consistency detection is also necessary. This is achieved by grouping speech segments and extracting the embedding sequences of the same speaker, constructing a time series in chronological order, and then calculating the cosine similarity of adjacent speech embedding vectors to form a temporal similarity matrix. This matrix can be used to assess the stability of the same speaker's speech performance across time periods. Combined with sliding window detection of abrupt changes and thresholding to determine the frequency of switching events, a speaker consistency score is generated. This mechanism ensures the continuity of customer identity in financial transactions, avoiding compliance risks arising from identity confusion or voiceprint recognition errors.

[0218] During the alternation of speech between the customer and customer service representatives, the rationality of the interaction also needs to be evaluated. This is achieved by detecting overlapping areas in the speech streams, recording start and end timestamps to obtain the actual overlap duration, and comparing it with the theoretical overlap duration of the reference text to calculate the time synchronization error rate. This, combined with a preset interaction rhythm threshold, generates an interaction rationality score. This score reflects whether there are frequent interruptions, excessive overlap, or unreasonable pauses between the customer and customer service representatives, thus measuring the fluency of the conversation and the customer experience.

[0219] Simultaneously, analyzing diversity features is also crucial. By extracting the voiceprint feature vectors of each speaker, a multi-dimensional speech feature sequence is constructed. The variance of each dimension's features is calculated and normalized, and then combined with preset weights to obtain a comprehensive variance value. This comprehensive variance value is compared with a threshold range to generate a diversity score, which quantifies the richness of speech features. The diversity score can reflect the changes in customer service representatives' intonation, rhythm, and emotional expression, avoiding monotonous and mechanical speech that could lead to a decline in customer experience.

[0220] Finally, the naturalness score, semantic consistency score, speaker consistency score, interaction rationality score, and diversity score are merged to generate a comprehensive quality score. Extreme value standardization unifies the dimensions of each score, and dynamic weights are obtained by combining the scene weight matrix and the speaker activity duration ratio. Exponentially decaying weights are applied to form a time-series weighted matrix, which is then weighted with the dynamic weight vector to obtain the initial comprehensive score. After removing low-confidence outliers using Monte Carlo confidence intervals, the remaining valid scores are geometrically averaged to form the final comprehensive quality score. This score can provide financial institutions with quantitative and traceable speech quality evaluation results.

[0221] In fintech scenarios, this process can be applied to the real-time monitoring of intelligent customer service systems. It automatically identifies instances of unnaturalness, semantic miscommunication, or identity verification errors in conversations between customer service representatives and customers, helping the system adjust its scripts or prompt for human intervention in a timely manner. Simultaneously, in remote financial consultations and compliance confirmation processes, the system can provide real-time feedback based on a comprehensive quality score, ensuring customers clearly understand compliance risk warnings or contractual points. This enhances the transparency of financial services and customer satisfaction, while reducing business risks and compliance vulnerabilities.

[0222] This embodiment addresses the numerical imbalance caused by different units by standardizing the scores across five dimensions. It ensures flexible and targeted weight allocation by combining the scene weight matrix and the speaker activity duration ratio. The introduction of a time factor through an exponential decay mechanism makes the influence of recent dialogue segments more significant on the final result. Outliers are eliminated using Monte Carlo confidence interval estimation, enhancing the stability and reliability of the scores. Finally, score fusion is achieved through geometric averaging to avoid excessive bias in the overall result caused by anomalies in a single dimension. The resulting comprehensive quality score more accurately and comprehensively reflects the overall performance of multi-speaker dialogue, providing reliable quantitative feedback for optimizing speech generation models and improving the evaluation and control capabilities of speech systems in applications such as fintech and healthcare.

[0223] In one embodiment, a multi-speaker dialogue speech analysis apparatus is provided, which corresponds one-to-one with the multi-speaker dialogue speech analysis method described in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the multi-speaker dialogue speech analysis device of the present invention. The modules include a speech acquisition module 10, an acoustic analysis module 20, a semantic alignment module 30, a speaker consistency module 40, an interaction analysis module 50, a diversity assessment module 60, and a quality fusion module 70. Detailed descriptions of each functional module are as follows:

[0224] The voice acquisition module 10 is used to acquire the voice of a multi-speaker dialogue to be analyzed;

[0225] The acoustic analysis module 20 is used to determine the naturalness score of the multi-speaker dialogue speech based on the acoustic features of the multi-speaker dialogue speech.

[0226] The semantic alignment module 30 is used to determine the semantic consistency score of the multi-speaker dialogue speech based on the speech embedding of the multi-speaker dialogue speech and the semantic embedding corresponding to the preset text.

[0227] Speaker consistency module 40 is used to determine the speaker consistency score of the multi-speaker dialogue speech based on the speaker embeddings corresponding to multiple speech segments of the same speaker in the multi-speaker dialogue speech.

[0228] The interaction analysis module 50 is used to determine the interaction rationality score of the multi-speaker dialogue speech based on the duration of speech alternation overlap.

[0229] The diversity assessment module 60 is used to determine the diversity score of the multi-speaker dialogue speech based on the variance of the speech features of the multi-speaker dialogue speech.

[0230] The quality fusion module 70 is used to fuse the naturalness score, the semantic consistency score, the speaker consistency score, the interaction rationality score, and the diversity score to obtain the comprehensive quality score of the multi-speaker dialogue speech.

[0231] In one embodiment, the acoustic analysis module 20 is specifically used for:

[0232] The acoustic features of the multi-speaker dialogue are extracted, and the acoustic features are standardized to generate standardized acoustic features.

[0233] Based on the standardized acoustic features, an acoustic feature vector containing temporal information is constructed;

[0234] The acoustic feature vector is input into a pre-trained naturalness scoring model to obtain the corresponding naturalness score;

[0235] The naturalness score is smoothed using a moving average or low-pass filter to generate the final naturalness score.

[0236] In one embodiment, the semantic alignment module 30 is specifically used for:

[0237] The speech embedding sequence of the multi-speaker dialogue is dynamically time-normalized and aligned with the semantic embedding sequence corresponding to the preset text to generate aligned speech embedding sequence and aligned semantic embedding sequence, and a temporal correspondence between speech frames and text units is established.

[0238] The aligned speech embedding sequence and the aligned semantic embedding sequence are normalized respectively to generate normalized speech embedding sequence and normalized semantic embedding sequence;

[0239] Based on the temporal correspondence, the frame-by-frame dot product of the normalized speech embedding sequence and the normalized semantic embedding sequence is determined to generate the original similarity score sequence.

[0240] The original similarity score sequence is filtered based on the voice activity detection results, and invalid scores with confidence levels lower than a preset confidence threshold are removed to generate the remaining valid similarity score sequence.

[0241] The dynamic weighting coefficient matrix is ​​determined based on the speech energy intensity at the corresponding position of the remaining valid similarity score sequence;

[0242] The remaining valid similarity score sequence is weighted and averaged using the dynamic weight coefficient matrix to generate the final semantic consistency score.

[0243] In one embodiment, the speaker consistency module 40 is specifically used for:

[0244] The speech segments in the multi-speaker dialogue are grouped based on the speaker identity identifier. Speaker embeddings are extracted from at least two speech segments of the same speaker. The speaker embeddings are then sorted in ascending order according to the start timestamp of the speech segments to construct a speaker embedding time series.

[0245] Determine the cosine similarity of speaker embedding vectors at adjacent positions in the speaker embedding time series, and generate a time series similarity matrix;

[0246] Based on the temporal similarity matrix, a stability index for speech feature changes across positions of the same speaker is determined;

[0247] The abrupt change points of the speech embedding vector in the speaker embedding time series are detected by a sliding window, and the total number of valid switching events is determined by combining the temporal similarity threshold.

[0248] The switching event frequency is determined based on the total number of valid switching events and the total number of interaction units in the multi-speaker dialogue.

[0249] Based on the stability index and the switching event frequency, a speaker consistency score is generated for the multi-speaker dialogue speech.

[0250] In one embodiment, the interactive analysis module 50 is specifically used for:

[0251] Detect overlapping regions in the speech streams of the multi-speaker dialogue and record the start and end timestamps of each overlapping region;

[0252] The actual speech overlap duration of the overlapping region is determined based on the start timestamp and the end timestamp.

[0253] Extract the theoretical duration of alternating and overlapping speech in the preset reference dialogue text;

[0254] The time synchronization error rate is determined based on the time difference between the actual speech overlap duration and the theoretical duration.

[0255] An interaction rationality score is generated based on the time synchronization error rate and the preset interaction rhythm threshold.

[0256] In one embodiment, the diversity assessment module 60 is specifically used for:

[0257] Extract the voiceprint feature vectors of each speaker in the multi-speaker dialogue speech to construct a multi-dimensional speech feature sequence;

[0258] Determine the variance value of each dimension of the feature in the multidimensional speech feature sequence, and generate a feature variance vector;

[0259] The feature variance vector is normalized to generate a normalized variance vector;

[0260] The normalized variance vector is weighted and fused according to preset weight coefficients to generate a comprehensive variance value;

[0261] Based on the comprehensive variance value and the preset diversity threshold range, a diversity score is generated for the multi-speaker dialogue speech.

[0262] In one embodiment, the quality fusion module 70 is specifically used for:

[0263] The naturalness score, semantic consistency score, speaker consistency score, interaction rationality score, and diversity score are respectively subjected to extreme value standardization to generate a standardized quality feature vector;

[0264] Based on the preset scene weight matrix and the proportion of active time of each speaker in the dialogue, the dynamic weight coefficient vectors of naturalness score, semantic consistency score, speaker consistency score, interaction rationality score and diversity score are determined through weighted processing.

[0265] Based on the temporal progression of the dialogue, an exponentially decaying weight is applied to the standardized quality feature vector to generate a time-weighted feature matrix that dynamically changes over time.

[0266] The dynamic weight coefficient vector and the time-series weighted feature matrix are weighted by dimension to obtain an initial value of the comprehensive quality score containing spatiotemporal features;

[0267] By estimating the confidence intervals using Monte Carlo methods, outlier data points with confidence levels below a preset threshold are removed from the initial comprehensive quality score to obtain the remaining valid scores.

[0268] The remaining valid scores are subjected to geometric averaging to generate a comprehensive quality score for the multi-speaker dialogue speech.

[0269] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a multi-speaker dialogue speech analysis method on the server side.

[0270] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a multi-speaker dialogue speech analysis method.

[0271] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0272] Acquire the multi-speaker dialogue audio to be analyzed;

[0273] Based on the acoustic features of the multi-speaker dialogue speech, a naturalness score of the multi-speaker dialogue speech is determined.

[0274] Based on the speech embedding of the multi-speaker dialogue speech and the semantic embedding corresponding to the preset text, the semantic consistency score of the multi-speaker dialogue speech is determined.

[0275] Based on the speaker embeddings corresponding to multiple speech segments of the same speaker in the multi-speaker dialogue speech, the speaker consistency score of the multi-speaker dialogue speech is determined.

[0276] Based on the duration of speech alternation and overlap in the multi-speaker dialogue, the interaction rationality score of the multi-speaker dialogue is determined.

[0277] The diversity score of the multi-speaker dialogue is determined based on the variance of the speech features of the multi-speaker dialogue.

[0278] The naturalness score, semantic consistency score, speaker consistency score, interaction rationality score, and diversity score are fused together to obtain the comprehensive quality score of the multi-speaker dialogue speech.

[0279] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0280] Acquire the multi-speaker dialogue audio to be analyzed;

[0281] Based on the acoustic features of the multi-speaker dialogue speech, a naturalness score of the multi-speaker dialogue speech is determined.

[0282] Based on the speech embedding of the multi-speaker dialogue speech and the semantic embedding corresponding to the preset text, the semantic consistency score of the multi-speaker dialogue speech is determined.

[0283] Based on the speaker embeddings corresponding to multiple speech segments of the same speaker in the multi-speaker dialogue speech, the speaker consistency score of the multi-speaker dialogue speech is determined.

[0284] Based on the duration of speech alternation and overlap in the multi-speaker dialogue, the interaction rationality score of the multi-speaker dialogue is determined.

[0285] The diversity score of the multi-speaker dialogue is determined based on the variance of the speech features of the multi-speaker dialogue.

[0286] The naturalness score, semantic consistency score, speaker consistency score, interaction rationality score, and diversity score are fused together to obtain the comprehensive quality score of the multi-speaker dialogue speech.

[0287] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0288] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0289] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0290] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for analyzing multi-speaker dialogue speech, characterized in that, Includes the following steps: Acquire the multi-speaker dialogue audio to be analyzed; Based on the acoustic features of the multi-speaker dialogue speech, a naturalness score of the multi-speaker dialogue speech is determined. Based on the speech embedding of the multi-speaker dialogue speech and the semantic embedding corresponding to the preset text, the semantic consistency score of the multi-speaker dialogue speech is determined. Based on the speaker embeddings corresponding to multiple speech segments of the same speaker in the multi-speaker dialogue speech, the speaker consistency score of the multi-speaker dialogue speech is determined. Based on the duration of speech alternation and overlap in the multi-speaker dialogue, the interaction rationality score of the multi-speaker dialogue is determined. The diversity score of the multi-speaker dialogue is determined based on the variance of the speech features of the multi-speaker dialogue. The naturalness score, semantic consistency score, speaker consistency score, interaction rationality score, and diversity score are combined to obtain the comprehensive quality score of the multi-speaker dialogue speech.

2. The method for analyzing multi-speaker dialogue speech as described in claim 1, characterized in that, Based on the acoustic features of the multi-speaker dialogue speech, a naturalness score is determined for the multi-speaker dialogue speech, including: The acoustic features of the multi-speaker dialogue are extracted, and the acoustic features are standardized to generate standardized acoustic features. Based on the standardized acoustic features, an acoustic feature vector containing temporal information is constructed; The acoustic feature vector is input into a pre-trained naturalness scoring model to obtain the corresponding naturalness score; The naturalness score is smoothed using a moving average or low-pass filter to generate the final naturalness score.

3. The method for analyzing multi-speaker dialogue speech as described in claim 1, characterized in that, Based on the speech embedding of the multi-speaker dialogue and the semantic embedding corresponding to the preset text, a semantic consistency score of the multi-speaker dialogue is determined, including: The speech embedding sequence of the multi-speaker dialogue is dynamically time-normalized and aligned with the semantic embedding sequence corresponding to the preset text to generate aligned speech embedding sequence and aligned semantic embedding sequence, and a temporal correspondence between speech frames and text units is established. The aligned speech embedding sequence and the aligned semantic embedding sequence are normalized respectively to generate normalized speech embedding sequence and normalized semantic embedding sequence; Based on the temporal correspondence, the frame-by-frame dot product of the normalized speech embedding sequence and the normalized semantic embedding sequence is determined to generate the original similarity score sequence. The original similarity score sequence is filtered based on the voice activity detection results, and invalid scores with confidence levels lower than a preset confidence threshold are removed to generate the remaining valid similarity score sequence. The dynamic weighting coefficient matrix is ​​determined based on the speech energy intensity at the corresponding position of the remaining valid similarity score sequence; The remaining valid similarity score sequence is weighted and averaged using the dynamic weight coefficient matrix to generate the final semantic consistency score.

4. The method for analyzing multi-speaker dialogue speech as described in claim 1, characterized in that, Based on the speaker embeddings corresponding to multiple speech segments of the same speaker in the multi-speaker dialogue, the speaker consistency score of the multi-speaker dialogue is determined, including: The speech segments in the multi-speaker dialogue are grouped based on the speaker identity identifier. Speaker embeddings are extracted from at least two speech segments of the same speaker. The speaker embeddings are then sorted in ascending order according to the start timestamp of the speech segments to construct a speaker embedding time series. Determine the cosine similarity of speaker embedding vectors at adjacent positions in the speaker embedding time series, and generate a time series similarity matrix; Based on the temporal similarity matrix, a stability index for speech feature changes across positions of the same speaker is determined; The abrupt change points of the speech embedding vector in the speaker embedding time series are detected by sliding window, and the total number of valid switching events is determined by combining the temporal similarity threshold. The switching event frequency is determined based on the total number of valid switching events and the total number of interaction units in the multi-speaker dialogue. Based on the stability index and the switching event frequency, a speaker consistency score is generated for the multi-speaker dialogue speech.

5. The method for analyzing multi-speaker dialogue speech as described in claim 1, characterized in that, Based on the duration of speech alternation overlap in the multi-speaker dialogue, an interaction rationality score is determined for the multi-speaker dialogue, including: Detect overlapping regions in the speech streams of the multi-speaker dialogue and record the start and end timestamps of each overlapping region; The actual speech overlap duration of the overlapping region is determined based on the start timestamp and the end timestamp. Extract the theoretical duration of alternating and overlapping speech in the preset reference dialogue text; The time synchronization error rate is determined based on the time difference between the actual speech overlap duration and the theoretical duration. An interaction rationality score is generated based on the time synchronization error rate and the preset interaction rhythm threshold.

6. The method for analyzing multi-speaker dialogue speech as described in claim 1, characterized in that, Based on the variance of the speech features of the multi-speaker dialogue, a diversity score is determined for the multi-speaker dialogue, including: Extract the voiceprint feature vectors of each speaker in the multi-speaker dialogue speech to construct a multi-dimensional speech feature sequence; Determine the variance value of each dimension of the feature in the multidimensional speech feature sequence, and generate a feature variance vector; The feature variance vector is normalized to generate a normalized variance vector; The normalized variance vector is weighted and fused according to preset weight coefficients to generate a comprehensive variance value; Based on the comprehensive variance value and the preset diversity threshold range, the diversity score of the multi-speaker dialogue speech is generated.

7. The method for analyzing multi-speaker dialogue speech as described in claim 1, characterized in that, The naturalness score, semantic consistency score, speaker consistency score, interaction rationality score, and diversity score are fused to obtain the comprehensive quality score of the multi-speaker dialogue speech, including: The naturalness score, semantic consistency score, speaker consistency score, interaction rationality score, and diversity score are respectively subjected to extreme value standardization to generate a standardized quality feature vector; Based on the preset scene weight matrix and the proportion of active time of each speaker in the dialogue, the dynamic weight coefficient vectors of naturalness score, semantic consistency score, speaker consistency score, interaction rationality score and diversity score are determined through weighted processing. Based on the temporal progression of the dialogue, an exponentially decaying weight is applied to the standardized quality feature vector to generate a time-weighted feature matrix that dynamically changes over time. The dynamic weight coefficient vector and the time-series weighted feature matrix are weighted by dimension to obtain an initial value of the comprehensive quality score containing spatiotemporal features; By estimating the confidence intervals using Monte Carlo methods, outlier data points with confidence levels below a preset threshold are removed from the initial comprehensive quality score to obtain the remaining valid scores. The remaining valid scores are subjected to geometric averaging to generate a comprehensive quality score for the multi-speaker dialogue speech.

8. A device for analyzing the speech of a multi-speaker dialogue, characterized in that, The multi-speaker dialogue speech analysis device includes: The voice acquisition module is used to acquire the voice of multi-speaker dialogues to be analyzed. An acoustic analysis module is used to determine the naturalness score of the multi-speaker dialogue speech based on the acoustic features of the multi-speaker dialogue speech. The semantic alignment module is used to determine the semantic consistency score of the multi-speaker dialogue speech based on the speech embedding of the multi-speaker dialogue speech and the semantic embedding corresponding to the preset text. The speaker consistency module is used to determine the speaker consistency score of the multi-speaker dialogue speech based on the speaker embeddings corresponding to multiple speech segments of the same speaker in the multi-speaker dialogue speech. An interaction analysis module is used to determine the interaction rationality score of the multi-speaker dialogue based on the duration of speech alternation overlap. A diversity assessment module is used to determine the diversity score of the multi-speaker dialogue speech based on the variance of the speech features of the multi-speaker dialogue speech. The quality fusion module is used to fuse the naturalness score, the semantic consistency score, the speaker consistency score, the interaction rationality score, and the diversity score to obtain the comprehensive quality score of the multi-speaker dialogue speech.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a multi-speaker dialogue speech analysis program stored in the memory and executable on the processor, wherein the multi-speaker dialogue speech analysis program, when executed by the processor, implements the steps of the multi-speaker dialogue speech analysis method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores an analysis program for multi-speaker dialogue speech, which, when executed by a processor, implements the steps of the multi-speaker dialogue speech analysis method as described in any one of claims 1-7.

Images