Large model-based voice dialogue system

Abstract

This invention relates to the field of voice interaction technology, and more particularly to a voice dialogue system based on a large model, comprising: a feature extraction module for acquiring samples containing multi-turn dialogue information and extracting multi-dimensional features; an alignment and processing module for achieving fine alignment through cross-modal alignment matrices and path search, and calculating a contextual consistency score based on role transition states; a stability calculation module for determining sample stability scores based on a teacher-student model mechanism; an anomaly assessment module for constructing a sample nearest neighbor graph based on semantic embedding vectors, calculating structural anomaly scores and sample retention values, and rescuing correctable samples by supplementing with nearest neighbor information; and a training module for selecting voice dialogue training samples according to sample retention values ​​and training the model. This invention provides high-quality samples for model training through a multi-dimensional quality assessment and data filtering mechanism, improving the accuracy and fluency of voice dialogue.

Description

Technical Field

[0001] This invention relates to the field of voice interaction technology, and more particularly to a voice dialogue system based on a large model. Background Technology

[0002] Large-scale voice dialogue systems demand high levels of naturalness, accuracy, and contextual coherence in responses, and heavily rely on large-scale, high-quality voice dialogue training data. Standard voice dialogue training samples typically need to include complete speech segments, accurate text annotations, clear turn boundaries, and key information such as speaker roles. In the field of multimodal data processing, existing technologies usually adopt a pipeline approach of independent governance of each modality followed by unified alignment. This involves configuring preprocessing tools and cleaning rules for each individual modality (speech, text, etc.), and then aligning and annotating them under a unified indexing system for large models to learn cross-modal correlations.

[0003] However, in the actual process of acquiring and constructing speech dialogue data, the acquired datasets are often filled with a large number of low-quality samples due to factors such as noisy acoustic environments and casual spoken expression. When the above-mentioned modal governance model is directly applied to speech dialogue training samples, its shortcomings become particularly prominent. Existing solutions still have room for improvement in terms of fine alignment between acoustics and text, multi-dimensional quality assessment of multi-turn dialogue context, and assessment of sample robustness to disturbances. This limits performance improvements in semantic understanding, multi-turn interaction fluency, and intent matching.

[0004] To address the consistency issue in multimodal data governance, referring to the Chinese patent document CN121144855B entitled "A Data Integration Governance System and Method for Multimodal Large Models," this solution aligns the collection timestamps of text, image, and speech data to a unified time index table by performing timestamp mapping processing. Then, semantic consistency detection is performed, identifying cross-modal semantic misalignments and generating anomaly markers by comparing text descriptions, image annotations, and speech content. Finally, correction labels are generated for misaligned items and updated to the corresponding modalities, forming a consistent training dataset.

[0005] While this scheme attempts to automate alignment and consistency verification, its alignment method is limited to coarse-grained mapping of acquisition timestamps, failing to delve into the fine-grained frame-level registration and dynamic correction between acoustic signals and text word sequences. In quality management of training samples for speech dialogues, it is typically impossible to perform frame-to-word level precise registration measurement and bias correction between acoustic features and text word temporal sequences; contextual consistency evaluation does not fully integrate dialogue structure features such as turn boundaries, role transitions, and slot constraints, resulting in a single evaluation dimension; and it lacks methods to assess the stability of samples against acoustic disturbances, making it difficult to filter low-quality samples that easily lead to training instability. Consequently, the trained speech dialogue model struggles to achieve ideal results in semantic understanding, multi-turn interaction fluency, and intent matching. Summary of the Invention

[0006] To address the technical problem that existing technologies lack multi-dimensional and detailed evaluation, making it difficult to ensure the acoustic, semantic, and contextual quality of training samples, this invention provides a voice dialogue system based on a large model.

[0007] This invention provides a large-scale model-based voice dialogue system, comprising: a feature extraction module, used to acquire voice dialogue training samples containing speech segments, text annotations, turn boundaries, and speaker roles; extract the acoustic coding vector of the speech segments; extract the word temporal vectors and semantic embedding vectors of the text annotations for the current and historical dialogue turns; and identify and construct slot constraint vectors for the text annotations; and an alignment and processing module, used to construct a cross-modal alignment matrix using the acoustic coding vectors and word temporal vectors and search for target matching paths; calculate the annotation registration score and generate corrected word temporal vectors; and combine the role transition flags determined by the turn boundaries and speaker roles, as well as the current and historical... The semantic embedding vector of the dialogue turn, the slot constraint vector, and the corrected word temporal vector are used to calculate the context consistency score; the stability calculation module is used to input the speech segment and the generated perturbed speech segment into the first reference model with frozen parameters and the second reference model in training state, respectively, and determine the sample stability score based on the difference in the output feature distribution; the anomaly evaluation module is used to perform local density analysis based on the annotation registration score, context consistency score, and sample stability score to generate a structural anomaly score, and calculate the sample retention value in combination with the structural anomaly score; the training module is used to select speech dialogue training samples according to the sample retention value and train the speech dialogue model.

[0008] This invention transforms unstructured speech and text into structured multidimensional feature vectors through a feature extraction module. By constructing a cross-modal alignment matrix and performing dynamic path search, it achieves millisecond-level precise registration of acoustic and text, effectively solving the problem of unmeasurable and uncorrectable acoustic-text boundary deviation in traditional solutions. Simultaneously, by integrating multidimensional information such as role switching, semantic coherence, temporal coherence, and intent expression, it can effectively identify low-quality samples caused by contextual logical breaks or missing intent. The stability calculation module constructs realistic perturbation scenarios such as speech rate changes and information loss to evaluate the stability of samples to acoustic interference, identifying unstable samples sensitive to noise. The anomaly assessment module detects structural outliers based on local density analysis and salvages correctable samples through a nearest neighbor information completion mechanism. Finally, it generates a sample retention value by integrating all evaluation dimensions, achieving an objective hierarchical judgment of sample quality. The training module uses this retention value to eliminate inferior data, ensuring high quality, high consistency, and high stability of the training set. This eliminates the negative interference of low-quality data on the optimization of large model parameters from the source, thereby improving the training efficiency, intent understanding accuracy, and multi-turn interaction fluency of the speech dialogue model.

[0009] Preferably, the alignment and processing module is specifically used for: calculating the cosine similarity between the acoustic coding vector and the word temporal vector to form the cross-modal alignment matrix; performing normalization processing on the cross-modal alignment matrix; and extracting the mean of the corresponding elements on the target matching path as the annotation registration score.

[0010] By constructing an alignment matrix using cosine similarity and performing probability normalization and dynamic path search, the degree of registration between acoustic signals and text annotations can be accurately assessed. Based on the searched target matching path, the original coarse word timestamps can be corrected to millisecond-level accuracy, eliminating the cumulative alignment errors caused by unclear pronunciation or changes in speech rate, and providing a high-precision temporal benchmark for subsequent temporal coherence analysis.

[0011] Preferably, the alignment and processing module is specifically used for: extracting entity nouns from the text annotations to construct slot constraint vectors; calculating the correlation between the historical semantic embedding vector corresponding to the previous dialogue round and the current semantic embedding vector to obtain semantic coherence confidence; extracting time intervals based on the corrected word temporal vectors to obtain temporal coherence features, extracting features from the slot constraint vectors to obtain intent expression features; and fusing the semantic coherence confidence, the temporal coherence features, and the intent expression features and mapping them to a preset interval to obtain the context consistency score.

[0012] By systematically integrating information from four dimensions—dialogue structure, semantic association, temporal coherence, and intent constraints—a multi-granularity context consistency evaluation mechanism is constructed. This mechanism can effectively identify low-quality samples caused by broken context logic, disordered role information, or missing intent, overcoming the problem of existing technologies having only one dimension for evaluating the quality of multi-turn dialogues.

[0013] Preferably, the perturbed speech segment includes a first perturbed speech segment and a second perturbed speech segment. The stability calculation module is specifically used for: applying speech rate perturbation to the speech segment to generate the first perturbed speech segment, and applying frequency band masking perturbation to generate the second perturbed speech segment; inputting the undisturbed speech segment into the first reference model to output a target feature distribution, and inputting the first perturbed speech segment and the second perturbed speech segment into the second reference model to output a predicted feature distribution; calculating the distribution divergence between the target feature distribution and the predicted feature distribution to obtain a divergence parameter, and determining the sample stability score based on the normalized value of the divergence parameter.

[0014] By perturbating in both the time and frequency domains, a realistic acoustic scenario with varying speech rate and missing information is constructed. By using a teacher model with frozen parameters and a student model in training state, and by minimizing the relative entropy divergence between feature distributions, the information redundancy and noise resistance of the samples themselves can be accurately captured, thereby effectively identifying low-quality samples that are sensitive to noise and prone to unstable model training.

[0015] Preferably, the anomaly assessment module is specifically used to: calculate the cosine distance between any two samples in a batch using the semantic embedding vector to form a distance matrix; and select semantically nearest neighbor samples based on the distance matrix to construct a sample nearest neighbor graph.

[0016] By constructing a sample nearest neighbor graph based on semantic embedding vectors and analyzing local density, it is possible to accurately identify structural outliers whose overall feature combinations in the semantic space are outside the mainstream distribution. The nearest neighbor determination method based on semantic cosine distance not only has a stronger adaptive ability to the differences in sample distribution within a batch, but also ensures the consistency of the nearest neighbor samples in terms of their true intent, laying a solid semantic foundation for subsequent slot feature completion with logical self-consistency.

[0017] Preferably, the anomaly assessment module is specifically used to: in response to the structural anomaly score not being greater than a preset anomaly threshold, extract the slot constraint vectors of all semantic nearest neighbor samples in the sample nearest neighbor graph as auxiliary constraint features; calculate the average value of the auxiliary constraint features of all nearest neighbor samples to obtain the expected feature vector; use the expected feature vector to perform constraint completion on the voice dialogue training samples, and recalculate the updated context consistency score.

[0018] By setting an anomaly threshold, low-quality samples are distinguished as recoverable and irrecoverable, avoiding the data waste caused by the blanket removal of all deviation samples. The expected feature vector completion mechanism based on the sample nearest neighbor graph can use the group statistical information of local clusters to smoothly repair missing or erroneous slot constraints, so that the semantic evaluation of recoverable samples can be corrected, increasing the possibility of their subsequent retention.

[0019] Preferably, the anomaly assessment module is specifically used to: perform a combined mapping on the annotation registration score, the context consistency score, the sample stability score, and the structural anomaly score using preset weights to generate a comprehensive score; perform normalization processing on the comprehensive score to obtain a normalized score; and use a nonlinear mapping function containing a logarithmic function to map the normalized score to a specified numerical range to obtain the sample retention value.

[0020] By using weighted combination, normalization, and nonlinear mapping, multi-dimensional and multi-dimensional quality assessment indicators are transformed into unified and intuitive sample retention values. The normalization operation effectively eliminates the risk of numerical dimension explosion caused by taking the reciprocal of outliers, ensures the absolute smoothness and mathematical rigor of the logarithmic mapping in the full-scale range, and balances the comprehensiveness of the assessment dimensions with the simplicity of the final decision, providing an objective basis for subsequent sample selection.

[0021] Preferably, the training module is specifically used for: in response to the sample retention value being lower than a preset retention threshold, extracting the identification information of low-quality labeled samples; performing data cleaning in the training sample library according to the identification information of the low-quality labeled samples to remove the corresponding voice dialogue training samples; and training the voice dialogue model using the remaining voice dialogue training samples after the removal operation.

[0022] By physically removing low-quality labeled samples, inferior data is effectively prevented from entering the model training stage, eliminating negative interference at the source. Using the selected high-quality, highly consistent, and highly stable samples to train large models can directly improve the model's performance metrics in semantic capture and response generation.

[0023] Preferably, the feature extraction module is specifically used for: inputting the speech segment into a pre-trained acoustic model to extract the acoustic coding vector; performing word segmentation on the text annotation using a word segmentation dictionary, and using a forced alignment model to add timestamp information to each word segment to generate the word temporal vectors for the current and historical dialogue rounds; inputting the text annotation into a pre-trained language model, and extracting the output feature vector as the semantic embedding vector for the current and historical dialogue rounds.

[0024] By systematically extracting acoustic, temporal, and global semantic features, unstructured speech and text signals are transformed into multi-level, computable feature vector sequences, providing a rich and complementary data foundation for subsequent multi-dimensional quality assessment. This overcomes the shortcomings of traditional schemes, such as single feature extraction dimensions and one-sided information representation.

[0025] The present invention also proposes an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the computer program, when executed by the processor, performs the following steps: Acquire speech dialogue training samples containing speech segments, text annotations, turn boundaries, and speaker roles; extract the acoustic coding vector of the speech segments, and the word temporal vector, semantic embedding vector, and slot constraint vector of the text annotations. An alignment matrix is ​​constructed to calculate the annotation registration score and correct the word temporal vector to generate a corrected word temporal vector. The context consistency score is calculated by combining the semantic embedding vector of the historical dialogue rounds, the slot constraint vector, the semantic embedding vector and the corrected word temporal vector. The speech segment and the perturbed speech segment are respectively input into a first reference model with frozen parameters and a second reference model in training state. The sample stability score is determined based on the difference in feature distribution output by the first reference model and the second reference model. Anomaly detection operations based on local density analysis are performed to generate structural anomaly scores, and sample retention values ​​are calculated based on these structural anomaly scores. The voice dialogue training samples are selected according to the sample retention values, and the voice dialogue model is trained.

[0026] The technical solution of the present invention has the following beneficial technical effects: This invention improves the accuracy of multimodal data alignment by comprehensively extracting multidimensional acoustic and textual features, calculating the annotation and registration scores of speech and text and performing temporal correction; and by combining multidimensional information such as turn boundaries, speaking roles and semantic slots to evaluate contextual consistency, ensuring the coherence of multi-turn dialogue logic and the accuracy of semantic expression.

[0027] Furthermore, sample stability is calculated using a perturbation-based teacher and student model mechanism, enhancing the samples' resistance to noise interference. Simultaneously, structural anomaly scores are calculated using the sample nearest neighbor graph, enabling the completion of slot constraints. By comprehensively calculating sample retention values ​​using multi-dimensional scoring metrics to eliminate low-quality samples, a high-quality, highly consistent, and highly stable data source is provided for model training. This avoids the negative interference of inferior data on parameter optimization, improving the training efficiency, intent understanding accuracy, and multi-turn interaction fluency of large-scale speech dialogue models. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the cross-modal alignment matrix; Figure 2 This is a diagram showing the comparison between the raw score and the weighted score. Figure 3 This is a schematic diagram comparing the experimental results. Detailed Implementation

[0029] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0030] This invention proposes a large-model-based voice dialogue system, comprising: a feature extraction module, an alignment and processing module, a stability calculation module, an anomaly evaluation module, and a training module.

[0031] The feature extraction module is used to obtain speech dialogue training samples containing speech segments, text annotations, turn boundaries, and speaking roles. This feature extraction module extracts the acoustic coding vectors of speech segments, as well as the word temporal vectors of text annotations, the semantic embedding vectors of the current and historical dialogue turns, and the slot constraint vectors from the speech dialogue training samples. As the data base for the subsequent multi-dimensional quality assessment process, the feature extraction module can transform the raw data of different modalities into standardized high-dimensional feature representations.

[0032] In an optional embodiment, the extraction process of the acoustic coding vector specifically involves: inputting a speech segment into a pre-trained acoustic model, which then outputs an acoustic coding vector. For example, the pre-trained acoustic model employs a Wav2vec2.0 network model, whose input is a speech segment waveform converted to a 16000Hz sampling rate. The network structure of this acoustic model includes a feature encoder composed of multiple one-dimensional convolutions and a context network composed of multiple transformer networks, and the output is a sequence of 768-dimensional acoustic coding vectors.

[0033] In an optional embodiment, the word-time vector generation process specifically involves: segmenting the text annotations using a pre-defined word segmentation dictionary, and adding timestamp information to each segmented word using a forced alignment model to generate word-time vectors. For example, a word segmentation toolkit with a preloaded dictionary size of 50,000 is used to segment the text, and the forced alignment module generates word start and end time information down to the 10ms level, encoding the words and their start and end timestamps into word-time vectors.

[0034] In an optional embodiment, the semantic embedding vector extraction process specifically involves: inputting text annotations into a pre-trained language model, and extracting the feature vector output by the pre-trained language model as the semantic embedding vector. For example, the pre-trained language model employs a BERT network model, whose input is a sequence of word segmentation results. The network structure is a bidirectional transformer-encoder network with 12 layers and a hidden layer size of 768. The feature vector of the first classification token (CLS) in the output layer of the last layer of the pre-trained language model is extracted as the 768-dimensional global semantic embedding vector.

[0035] In an optional embodiment, the construction process of the slot constraint vector is as follows: based on a preset dialogue intent slot ontology specification, entity nouns are extracted from the text annotations, and a fixed-length flag bit matrix is ​​constructed as the slot constraint vector. For example, entity word matching is performed based on the ontology specification of 50 common dialogue slot types such as location, time, and action, and a binary flag bit matrix with a fixed length of 256 is constructed using one-hot encoding, with 1 and 0 respectively representing whether the text segment contains the corresponding key slot entity noun.

[0036] Thus, through the aforementioned feature extraction mechanism, unstructured speech signals and text annotations can be systematically transformed into a multi-level, computable sequence of feature vectors. Specifically, the acoustic coding vector meticulously characterizes the underlying acoustic properties of speech, the word temporal vector provides the initial alignment benchmark between the text and the timeline, the semantic embedding vector captures the global contextual meaning of the text, and the slot constraint vector injects task-oriented structured prior knowledge. This provides a solid data foundation for subsequent steps to accurately evaluate the degree of acoustic-text registration, assess the contextual consistency of multi-turn dialogues, and detect the stability of samples, effectively overcoming the shortcomings of traditional schemes, such as single-dimensional feature extraction and one-sided information representation.

[0037] The alignment and processing module is used to construct a cross-modal alignment matrix using acoustic coding vectors and word temporal vectors, to calculate the annotation registration score and correct the word temporal vectors, generating corrected word temporal vectors.

[0038] In an optional embodiment, the above process specifically involves: calculating the cosine similarity between each time step feature of the acoustic coding vector and each semantic node feature of the word temporal vector, and arranging all the calculated cosine similarities to form a two-dimensional cross-modal alignment matrix; performing a Softmax function to normalize the cross-modal alignment matrix in the column direction; using a dynamic time warping algorithm to search for the maximum likelihood probability matching path in the normalized cross-modal alignment matrix, extracting the mean of the corresponding elements on the maximum likelihood probability matching path as the annotation registration score, and determining the start and end times of each semantic node according to the maximum likelihood probability matching path to update the timestamp, thereby obtaining the corrected word temporal vector.

[0039] For example, such as Figure 1 The diagram shows a heatmap of the cross-modal alignment matrix constructed from the acoustic frame index and semantic node index in this embodiment. The extracted acoustic coding vector contains M time steps with a feature dimension of 768, and the word temporal vector contains N semantic nodes, also mapped to a feature dimension of 768. Cosine similarity is calculated point-by-point through dot product operations and L2 norm normalization to generate a two-dimensional cross-modal alignment matrix of size M×N. To highlight highly relevant matching nodes, a Softmax function with a temperature coefficient of 0.5 is used to perform normalization along the N-dimensional column direction, mapping all similarity scores to a probability range of 0 to 1. In the matching search and time correction stages, the underlying layer uses the standard Euclidean distance function to construct the matching cost matrix of the dynamic time warping algorithm. Within the M×N normalized matrix, dynamic programming is used to search for the diagonal or horizontal transition trajectory with the minimum cumulative cost. This trajectory is the maximum likelihood probability matching path. The probability values ​​at all coordinate positions constituting this trajectory are summed and divided by the path length; the result is used as the annotation registration score of the speech dialogue training sample. Based on the maximum likelihood probability matching path, the start frame sequence number and end frame sequence number of each word index on the acoustic feature time axis are found in reverse. The original timestamp information is then rewritten according to the rule that time equals frame number multiplied by 10ms, corrected to the millisecond level, thereby generating the corrected word time sequence vector.

[0040] Thus, by constructing a cross-modal alignment matrix and performing probability normalization and dynamic path search, the degree of registration between acoustic signals and text annotations can be evaluated, effectively solving the problem that traditional schemes struggle to measure and correct acoustic-text boundary deviations. Based on the target matching path obtained through the search, the original coarse word timestamps are corrected to millisecond-level precision, eliminating accumulated alignment errors caused by unclear pronunciation or changes in speech rate, and providing a reliable foundation for the subsequent calculation of temporal coherence features in the contextual consistency score.

[0041] The alignment and processing module is also used to combine turn boundaries, speaking roles, semantic embedding vectors, corrected word temporal vectors, and slot constraint vectors to calculate the context consistency score.

[0042] In an optional embodiment, the process is as follows: based on the turn boundary and the speaking role, it is determined whether the speaking role of the current dialogue turn has changed from that of the previous dialogue turn, a role change flag is generated, the historical semantic embedding vector of the previous dialogue turn is concatenated with the current semantic embedding vector and input into the multilayer perceptron network, the semantic coherence confidence score is output, the temporal coherence feature is obtained by calculating the temporal interval between adjacent words based on the corrected word temporal vector, and the intention expression feature is obtained by extracting features from the slot constraint vector. The role change flag, semantic coherence confidence score, temporal coherence feature and intention expression feature are input into a preset scoring network for weighted summation, and the summation result is mapped to a preset interval through an activation function to obtain the context consistency score.

[0043] For example, in the role switching detection stage, the speaking role identifier is compared between previous and subsequent rounds. If the previous round's speaking role is A and the current round's speaking role is B, a role switching flag with a value of 1 is generated; otherwise, the flag is set to 0. In the semantic interaction layer, the previous round's historical semantic embedding vector (both dimensions 768) and the current round's semantic embedding vector are concatenated along the channel dimension to form a 1536-dimensional fusion vector, which is then fed into a multilayer perceptron network. The input to this network is the 1536-dimensional concatenated fusion vector. The network structure of the multilayer perceptron is a feedforward neural network containing two fully connected layers, with 512 and 1 nodes respectively. The hidden layers use the ReLU activation function, and the output layer outputs a floating-point value between 0 and 1 using the Sigmoid operator, serving as the semantic coherence confidence score. For temporal feature extraction, a one-dimensional sequence is constructed as a temporal coherence feature by extracting the time stamps between consecutive word blocks within the corrected word temporal vector. For intent expression feature extraction, a 256-dimensional slot constraint vector is input into a feature extraction network containing a one-dimensional convolutional layer. This network compresses and reduces the dimensionality of the slot constraint vector, outputting 64-dimensional intent expression features. The four different dimensions of features—role transition marker, semantic coherence confidence, temporal coherence features, and intent expression features—are mapped to the same scalar dimension through fully connected layers. The scoring network then performs a weighted fusion of these four features, assigning a weight of 0.1 to the role transition marker, 0.5 to the semantic coherence confidence, 0.2 to the temporal coherence, and 0.2 to the intent expression features. Weighted summation is performed using matrix multiplication, and the summation result is non-linearly constrained to the 0-1 interval using a sigmoid activation function to obtain the context consistency score.

[0044] Thus, by constructing a multi-dimensional, multi-granular contextual consistency assessment mechanism, information from four dimensions—dialogue structure, semantic relevance, temporal coherence, and intent constraints—is integrated. Role-switching markers explicitly model structural switching events in multi-turn dialogues; semantic coherence confidence captures the quality of semantic flow across turns through deep networks; temporal coherence features extract pause and speech rate patterns using high-precision corrected temporal vectors; and intent expression features condense high-dimensional slot constraints into compact intent representations. Through adaptive weighted fusion of the scoring network, the output contextual consistency score can effectively identify low-quality samples caused by contextual logical breaks, role confusion, or missing intent, effectively overcoming the shortcomings of existing technologies in assessing the quality of multi-turn dialogues with a single dimension.

[0045] The stability calculation module is used to input the speech segment and the generated perturbed speech segment into the first reference model and the second reference model respectively, and determine the sample stability score based on the difference in feature distribution output by the first reference model and the second reference model.

[0046] In an optional embodiment, the process specifically involves: applying a preset speech rate perturbation to a speech segment, generating a first perturbed speech segment through temporal stretching; applying a preset frequency band masking perturbation to the speech segment, generating a second perturbed speech segment through masking within a preset frequency range; inputting the undisturbed original speech segment into a first reference model with frozen parameters, outputting a target feature distribution; inputting the first and second perturbed speech segments into a second reference model under training, respectively, outputting a first and second predicted feature distributions; performing global pooling dimensionality reduction on the target feature distribution, the first predicted feature distribution, and the second predicted feature distribution; calculating the relative entropy between the processed target feature distribution and the first and second predicted feature distributions, respectively; and summing and averaging the relative entropies to obtain a divergence parameter; taking the reciprocal of the divergence parameter and normalizing it, using the normalized value as the sample stability score.

[0047] For example, for a 5-second original speech segment as input, a temporal synchronous overlay algorithm is used to uniformly increase the speech rate by 1.1 times while maintaining the pitch, resulting in a first perturbed speech segment with a total duration of 4.54 seconds. Simultaneously, at the Mel spectrogram layer level, a 200Hz bandwidth mask is applied to the original segment within the 1000Hz to 1500Hz center frequency range, completely clearing the pixel values ​​in the corresponding frequency domain, thus synthesizing a second perturbed speech segment with bandwidth loss. Both the first and second reference models are deployed as transformer-based network models containing 24 layers of self-attention coding blocks. The input to the first reference model is the undisturbed original speech segment, its network parameters have stopped updating, and its output is the target feature distribution. The input to the second reference model is the first and second perturbed speech segments, and its network is in a training state where it is iteratively calculating gradients, outputting the first and second predicted feature distributions. The specific mechanism for the second reference model being in the training state is as follows: the first reference model with frozen parameters serves as the teacher model, and the second reference model serves as the student model. The original, undisturbed speech segment is input as the first reference model for the teacher model, and the first and second perturbed speech segments are input as the second reference model for the student model. The parameters of the student model are updated by minimizing the relative entropy divergence between the feature distributions output by the teacher and student models for the same perturbed speech segment as the loss function. Global average pooling is used to uniformly reduce the time axis length of the target feature distribution, the first predicted feature distribution, and the second predicted feature distribution to a 256-dimensional feature vector. The relative entropy formula is used to separately measure the two relative entropies between the target feature distribution and the first predicted feature distribution, and between the target feature distribution and the second predicted feature distribution. The divergence parameter is obtained by summing and averaging the two relative entropies. The reciprocal of the divergence parameter is taken and mapped to the interval between 0 and 1 through a predefined normalization scaling mechanism. This value is used as the sample stability score representing the sample's resistance to acoustic noise variation.

[0048] Thus, by comparing the feature distribution differences between the first reference model with frozen parameters and the second reference model with continuous learning on the original samples and perturbed samples, we can accurately capture the information redundancy and noise resistance of the samples themselves. The smaller the divergence parameter, the more consistent the feature representation of the sample remains under different perturbations, and the higher the stability of the sample. This can effectively identify low-quality samples that are sensitive to noise and easily lead to unstable model training, providing a basis for constructing a highly stable training dataset.

[0049] The anomaly assessment module is used to perform node operations based on the annotation registration score, context consistency score, and sample stability score to generate a structural anomaly score.

[0050] In an optional embodiment, the process specifically involves: using the semantic embedding vectors of each speech dialogue training sample within a batch as feature vectors; calculating the cosine distance between the feature vectors of any two speech dialogue training samples within the batch to form a distance matrix; for the target sample, selecting the top preset number of speech dialogue training samples with the smallest distance from each other based on the distance matrix as nearest neighbor samples; constructing a sample nearest neighbor graph by connecting the target sample and its nearest neighbor samples through edges; calculating the average distance between the target sample and all its nearest neighbor samples as a first offset; calculating the average distance between each pair of all its nearest neighbor samples as a second offset; and using the ratio of the first offset to the second offset as the structural anomaly score of the target sample.

[0051] For example, the number of voice dialogue training samples in a single processing round is fixed at 256. The previously calculated annotation registration score, context consistency score, and sample stability score of each voice dialogue training sample are concatenated in sequence to form a three-dimensional feature vector, which is used as the coordinate representation of the voice dialogue training sample in high-dimensional space. Tensor quantization is used to accelerate pairwise pairing. The mutual distance between the three-dimensional feature vectors of all 256 samples is calculated in batches using the Euclidean norm formula, generating a global symmetric distance matrix of size 256×256. For the target sample, the corresponding row is extracted based on the distance matrix, the values ​​are arranged in ascending order, and the first to fifth distance values ​​(excluding itself) are selected, i.e., the preset number is set to 5. The target sample is connected to the 5 selected highly similar nearest neighbor samples through edges to form a sample nearest neighbor graph with a star-shaped connection structure. Further, the sum of the Euclidean distances from the target node to these 5 nearest neighbor nodes is calculated and divided by 5 to obtain the first offset. The average Euclidean distance of a total of 10 internal combinations between the 5 nearest neighbor samples is calculated to obtain the second offset. Divide the first offset by the second offset, and the resulting ratio is taken as the structural anomaly score of the target sample. When the ratio is greater than 1, it indicates that the speech dialogue training sample is outside the center of its local cluster.

[0052] The anomaly assessment module is also used to calculate the sample retention value by combining the structural anomaly score. When the structural anomaly score is not greater than the anomaly threshold, the slot constraint vector is completed using nearest neighbor samples to obtain the updated context consistency score; otherwise, the original context consistency score is used as the updated context consistency score.

[0053] In an optional embodiment, the process of slot constraint completion and context consistency score update is as follows: In response to the structural anomaly score not being greater than a preset anomaly threshold, auxiliary constraint features of all neighboring samples in the sample's nearest neighbor graph are extracted, the average value of the auxiliary constraint features of all neighboring samples is calculated to obtain the desired feature vector, and the slot constraint vector of the target sample is weighted and fused to complete the slot constraint vector using the desired feature vector to obtain the soft-constraint completed slot constraint vector. The soft-constraint completed slot constraint vector replaces the original slot constraint vector of the target sample, and after feature extraction, it is re-input into the scoring network for calculating the context consistency score to calculate the updated context consistency score. When the structural anomaly score is greater than the anomaly threshold, the original context consistency score is directly used as the updated context consistency score.

[0054] For example, the anomaly threshold is set to 3. If the structural anomaly score of the target sample is 2.5, satisfying the condition of not exceeding 3, then the 256-dimensional slot constraint vectors corresponding to the 5 semantic nearest neighbor nodes associated with the sample in the nearest neighbor graph are extracted. Point-to-point addition is performed and the arithmetic mean is calculated by dividing by 5 to obtain a 256-dimensional continuous value expected feature vector representing the general state of the surrounding area. The first weight parameter is set to 0.7 to represent the weight of the group correction influence, and the second weight parameter is set to 0.3 to retain the weight of the sample's own characteristics. Each element of the expected feature vector is uniformly multiplied by 0.7, and the original slot constraint vector of the target sample is uniformly multiplied by 0.3. The two are superimposed and fused to generate a slot constraint vector with soft constraint completion. The completed slot constraint vector is fed into a 64-dimensional convolutional compression layer for the same processing, and then passed back to the scoring network for re-inference to obtain the updated context consistency score. If the structural anomaly score of the target sample reaches 3.5, exceeding the anomaly threshold of 3, the interpolation repair action is abandoned, and the original context consistency score is used as the updated context consistency score unchanged.

[0055] The anomaly assessment module is also used to perform a combined mapping on the annotation registration score, the updated context consistency score, the sample stability score, and the structural anomaly score using preset weights to generate a comprehensive score. The comprehensive score is then mapped to a specified numerical range using a preset nonlinear transformation model to obtain the sample retention value.

[0056] For example, the four weighting coefficients are set to 0.3, 0.3, 0.3, and 0.1, respectively. The structural anomaly score is a negative indicator, and its reciprocal needs to be calculated beforehand to make it more positive. The values ​​are set as follows: annotation registration score 0.85, updated context consistency score 0.82, sample stability score 0.88, and the reciprocal of the structural anomaly score of 2.5 is 0.4. The distribution of the above evaluation feature scores and weighted scores is then compared. Figure 2 As shown, the arithmetic unit performs linear combination accumulation to obtain a comprehensive score. Performing Min-Max normalization and mapping to the [0, 1] interval yields... Then input includes operators The nonlinear transformation model is used for internal transformation, mapping to a percentage range of 0 to 100 to obtain the retained sample values.

[0057] in, This represents the retained values ​​of the samples obtained after mapping. This represents the comprehensive score after normalization; where... This operation is to smooth the calculation and avoid issues with the normalized overall score. When the value is 0, the logarithmic calculation is meaningless. The constant 100 is a scaling factor used to map the full-scale result of the final calculation to a specified numerical range of 0 to 100 in percentile format.

[0058] Thus, by setting an anomaly threshold, low-quality samples are distinguished as recoverable or irrecoverable, avoiding the data waste caused by the indiscriminate removal of all deviating samples. Based on the expected feature vector completion mechanism of the sample nearest neighbor graph, the group statistical information of local clusters is used to smoothly repair missing or erroneous slot constraints, so that the context consistency assessment of recoverable samples can be corrected, increasing their likelihood of being retained. Through weighted combination and nonlinear mapping, the evaluation scores of the four dimensions are transformed into a unified and intuitive sample retention value, providing an objective decision basis for subsequent sample selection, and taking into account both the comprehensiveness of the evaluation dimensions and the operability of the final decision.

[0059] The training module is used to filter speech dialogue training samples and train the speech dialogue model according to the sample retention value. Specifically, it is used to extract the labeling information of low-quality labeled samples in response to the sample retention value being lower than the preset retention threshold; perform data cleaning in the training sample library according to the labeling information of low-quality labeled samples to remove the corresponding speech dialogue training samples; and use the remaining speech dialogue training samples after the removal operation to train the speech dialogue model.

[0060] In an optional embodiment, the sample screening process specifically involves: determining whether the sample retention value is lower than a preset retention threshold; if the sample retention value is lower than the retention threshold, then determining that the corresponding voice dialogue training sample is a low-quality labeled sample; performing a data cleaning operation in the training sample library based on the identification information of the low-quality labeled sample to remove the corresponding voice dialogue training sample; and using the remaining high-quality voice dialogue training samples after removing the low-quality samples to train the voice dialogue model.

[0061] For example, the retention threshold is set to 60 points. If the retention value of a damaged sample is 42.5 points, which is lower than the 60-point threshold, a low-quality flag is assigned to it. The executor obtains the unique identifier of the sample carrying this flag, locates the corresponding index table block in the underlying distributed file system, issues a deletion command to unbind the audio file and the corresponding annotation file address, and erases the memory handle. The selected high-quality samples are instantiated into a dataset class and passed to the data loader for batch data loading. Using the third-stage memory optimization strategy of the distributed training framework, the large language model with 7 billion parameters is fine-tuned and trained. During training, the AdamW optimizer is used with a learning rate of 2×10⁻⁶. -5 The network weights of the large language model are updated to minimize the autoregressive cross-entropy loss function. After the model converges and the training weights are saved, the inference function is called on the GPU deployment server to receive the real-time audio stream captured by the user's microphone, perform streaming decoding, and output the corresponding text response. Alternatively, a human-like speech synthesis tool can be used to convert the generated text into speech before outputting it, realizing end-to-end voice dialogue interaction.

[0062] In this way, by setting a retention threshold and physically removing low-quality labeled samples, it is possible to effectively prevent poor-quality data with high acoustic noise, serious text alignment deviation, or broken contextual logic from entering the model training stage. This eliminates the negative interference of low-quality data on model parameter optimization from the source. By using the selected high-quality, highly consistent, and highly stable samples to train a large language model with hundreds of billions of parameters, it is possible to improve the model's training efficiency, the accuracy of intent understanding, and the fluency of multi-turn interactions.

[0063] To verify the effectiveness of the data selection and multimodal correction scheme, the experiment used a round-based two-person voice interaction dataset as the training set, and the test set was independently divided into 10,000 rounds of dialogue segments. The downstream evaluation platform adopted a generative dialogue model with a standard bidirectional encoder-decoder structure. The experimental environment was uniformly configured with the same graphics card hardware and driver environment. During model training, the batch size was uniformly set to 256, and the global learning rate remained constant. The evaluation metrics focused on the slot F1 score, which reflects semantic capture ability, and the ROUGE text matching score and BLEU generation coherence score, which measure the quality of text responses.

[0064] Group 1, serving as the baseline control group, did not use the data filtering and feature completion process described in this paper at all; it only input the original phonetic-character timestamp aligned corpus.

[0065] The second ablation group disables the sample retention value safety cutoff function, meaning it does not remove any low-quality data.

[0066] The third ablation group retains the cleaning function but blocks the slot soft constraint completion mechanism based on the expected value of nearest neighbor nodes.

[0067] Group 4 deployment includes a complete processing solution that includes alignment correction of structural anomalies, scoring, and precise interception of low-quality data.

[0068] Data collected during the experiment, and comparisons of experimental results are as follows: Figure 3 As shown.

[0069] Group 1, without using the filtering and completion process, only the raw corpus was input. The slot F1 score was 76.5%, the ROUGE score was 42.3%, and the BLEU score was 18.6%.

[0070] Group 2, without removing any low-quality data and with the sample retention value safety truncation function turned off, the slot F1 score was 79.8%, the ROUGE score was 44.1%, and the BLEU score was 19.5%.

[0071] Group 3 retains the cleaning function but blocks the slot soft constraint completion mechanism. Slot F1 score is 81.2%, ROUGE score is 45.6%, and BLEU score is 20.4%. Group 4, complete solution, slot F1 score 86.4%, ROUGE score 49.2%, BLEU score 23.1%.

[0072] Combination Figure 3 The experimental results shown in the bar chart demonstrate that the present invention, which employs a complete processing scheme, significantly outperforms the baseline model and all ablation groups in various evaluation metrics, including slot F1 score, ROUGE score, and BLEU score, fully proving the effectiveness of this data screening and multimodal correction scheme.

[0073] It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept, and these all fall within the scope of protection of this invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A voice dialogue system based on a large model, characterized in that, include: The feature extraction module is used to acquire speech dialogue training samples containing speech segments, text annotations, turn boundaries, and speaking roles; extract the acoustic coding vector of the speech segments; extract the word temporal vector and semantic embedding vector of the text annotations for the current and historical dialogue turns; and identify and construct slot constraint vectors for the text annotations. The alignment and processing module is used to construct a cross-modal alignment matrix using the acoustic coding vector and word temporal vectors and search for target matching paths; calculate the annotation registration score and generate corrected word temporal vectors; and calculate the context consistency score by combining the role transition flags determined by the turn boundaries and speaking roles, the semantic embedding vectors of the current and historical dialogue turns, the slot constraint vectors, and the corrected word temporal vectors. The stability calculation module is used to input the speech segment and the generated perturbed speech segment into the first reference model with frozen parameters and the second reference model in the training state, respectively, and determine the sample stability score based on the difference in the output feature distribution; the anomaly evaluation module is used to perform local density analysis based on the annotation registration score, context consistency score and sample stability score to generate a structural anomaly score, and calculate the sample retention value in combination with the structural anomaly score. The training module is used to filter voice dialogue training samples according to the sample retention values ​​and train the voice dialogue model.

2. The large-model-based voice dialogue system according to claim 1, characterized in that, The alignment and processing module is specifically used for: Calculate the cosine similarity between the acoustic coding vector and the word temporal vector to form the cross-modal alignment matrix; perform normalization processing on the cross-modal alignment matrix; extract the mean of the corresponding elements on the target matching path as the annotation registration score.

3. The large-model-based voice dialogue system according to claim 1, characterized in that, The alignment and processing module is specifically used for: Extract entity nouns from the text annotations to construct slot constraint vectors; The semantic coherence confidence score is obtained by calculating the correlation between the historical semantic embedding vector corresponding to the previous dialogue round and the current semantic embedding vector. Based on the corrected word temporal vector, the temporal interval is extracted to obtain temporal coherence features, and features are extracted from the slot constraint vector to obtain intention expression features; The semantic coherence confidence score, the temporal coherence feature, and the intent expression feature are fused and mapped to a preset interval to obtain the context consistency score.

4. The large-model-based voice dialogue system according to claim 1, characterized in that, The disturbed speech segment includes a first disturbed speech segment and a second disturbed speech segment, and the stability calculation module is specifically used for: A speech rate perturbation is applied to the speech segment to generate the first perturbated speech segment, and a frequency band masking perturbation is applied to generate the second perturbated speech segment; The undisturbed speech segment is input into the first reference model to output the target feature distribution, and the first perturbed speech segment and the second perturbed speech segment are input into the second reference model to output the predicted feature distribution. The divergence between the target feature distribution and the predicted feature distribution is calculated to obtain the divergence parameter, and the sample stability score is determined based on the normalized value of the divergence parameter.

5. The large-model-based voice dialogue system according to claim 1, characterized in that, The anomaly assessment module is specifically used for: The cosine distance between any two samples in a batch is calculated using the semantic embedding vector to form a distance matrix; semantic nearest neighbor samples are selected based on the distance matrix to construct a sample nearest neighbor graph.

6. The large-model-based voice dialogue system according to claim 5, characterized in that, The anomaly assessment module is specifically used for: In response to the structural anomaly score not being greater than a preset anomaly threshold, the slot constraint vectors of all semantic nearest neighbor samples in the sample nearest neighbor graph are extracted as auxiliary constraint features. The desired feature vector is obtained by averaging the auxiliary constraint features of all nearest neighbor samples. The expected feature vector is used to perform constraint completion on the voice dialogue training samples, and the updated context consistency score is recalculated.

7. The large-model-based voice dialogue system according to claim 1, characterized in that, The anomaly assessment module is specifically used for: A comprehensive score is generated by performing a combined mapping on the annotation registration score, the context consistency score, the sample stability score, and the structural anomaly score using preset weights; the comprehensive score is then normalized to obtain a normalized score; and the normalized score is mapped to a specified numerical range using a nonlinear mapping function containing a logarithmic function to obtain the sample retention value.

8. The large-model-based voice dialogue system according to claim 1, characterized in that, The training module is specifically used for: In response to the sample retention value being lower than a preset retention threshold, the identification information of the low-quality labeled sample is extracted; Based on the identification information of the low-quality labeled samples, data cleaning is performed in the training sample library to remove the corresponding voice dialogue training samples; The speech dialogue model is trained using the remaining speech dialogue training samples after the elimination operation.

9. The large-model-based voice dialogue system according to claim 1, characterized in that, The feature extraction module is specifically used for: The acoustic coding vector is obtained by inputting the speech segment into a pre-trained acoustic model. The text annotations are segmented using a word segmentation dictionary. A forced alignment model is used to add timestamp information to each segment to generate word time sequence vectors for the current and previous dialogue rounds. The text annotations are then input into a pre-trained language model, and the output feature vectors are extracted as semantic embedding vectors for the current and previous dialogue rounds.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, it performs the following steps: Acquire speech dialogue training samples containing speech segments, text annotations, turn boundaries, and speaker roles; extract the acoustic coding vector of the speech segments, and the word temporal vector, semantic embedding vector, and slot constraint vector of the text annotations. An alignment matrix is ​​constructed to calculate the annotation registration score and correct the word temporal vector to generate a corrected word temporal vector. The context consistency score is calculated by combining the semantic embedding vector of the historical dialogue rounds, the slot constraint vector, the semantic embedding vector and the corrected word temporal vector. The speech segment and the generated perturbed speech segment are respectively input into a first reference model with frozen parameters and a second reference model in training state. The sample stability score is determined based on the difference in feature distribution output by the first reference model and the second reference model. Anomaly detection operations based on local density analysis are performed to generate structural anomaly scores, and sample retention values ​​are calculated based on these structural anomaly scores. The voice dialogue training samples are selected according to the sample retention values, and the voice dialogue model is trained.

Images