Deep learning-based voice diagnostic call system, method, and storage medium

Abstract

The application discloses a speech diagnosis calling system and method based on deep learning and a storage medium, and comprises the following steps: step one, collecting speech signals and performing pretreatment; step two, performing disturbance processing; step three, inputting the disturbed speech information into an improved HuBERT model, introducing a dynamic semantic mask module and a gated memory fusion module, and extracting a speech context representation vector; step four, inputting the speech context representation vector into a time sequence convolution attention network, and extracting a frame-level time context feature; step five, performing label classification based on a DP-means clustering algorithm; step six, generating a structured diagnosis result based on a frame-level speech abnormal label sequence; and step seven, sending the structured diagnosis result to a speech calling platform, so that speech diagnosis and calling service linkage are realized. Through the combination of the improved HuBERT model and the DP-means clustering algorithm, the application realizes accurate extraction of speech abnormal labels and calling linkage.

Description

Technical Field

[0001] The present invention relates to the technical field of speech processing and intelligent diagnosis, and particularly to a speech diagnosis call system, method and storage medium based on deep learning. Background Art

[0002] With the development of intelligent voice communication technology, voice call devices have been widely deployed in various scenarios such as customer service, remote interaction, and emergency response. To improve the user experience and interaction efficiency, voice call systems increasingly rely on the automatic analysis of voice content, anomaly detection, and dynamic response strategies. However, current voice call devices still face a series of technical challenges in the process of processing natural voice streams:

[0003] Due to factors such as network fluctuations, background noise, or unclear user pronunciation, the collected voice data has problems of unstructuredness, strong perturbation, and low signal-to-noise ratio. Traditional methods based on static feature extraction and simple classification models are difficult to adapt to the variable voice states in real conversations; mainstream deep voice models generally ignore the dynamic characteristics of frame-level voice perturbation in the extraction of context information, and the masking mechanism lacks adaptability, resulting in redundant or distorted context representations; in addition, when identifying anomalies in continuous voice sequences, fixed-category clustering methods are difficult to handle complex patterns such as semantic mutations and sudden anomalies in natural language, lack dynamic adjustment capabilities, and limit the accuracy of subsequent diagnosis and response strategies. Existing voice anomaly detection results are mostly presented as unstructured labels or overall scores, which are difficult to accurately dock with the scheduling module of the voice call platform, unable to form linkage rules based on time indexes, and restrict the intelligence and automation levels of system responses.

[0004] Therefore, how to provide a speech diagnosis call system, method and storage medium based on deep learning is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0005] An object of the present invention is to propose a speech diagnosis call system, method and storage medium based on deep learning. The present invention combines an improved HuBERT model introducing a dynamic semantic masking module and a gated memory fusion module with a DP-means clustering algorithm with strong adaptive clustering ability to realize frame-level semantic modeling and temporal clustering of the perturbed enhanced voice signal, thereby accurately extracting frame-level voice anomaly labels and generating structured diagnosis results, and finally realizing the linkage call with the voice call platform, having the remarkable advantages of high semantic extraction accuracy, strong classification robustness, and being applicable to complex call scenarios.

[0006] The speech diagnosis call method based on deep learning according to an embodiment of the present invention includes the following steps:

[0007] Step 1: Collect the user's voice signal during the call interaction process, and preprocess the voice signal to obtain a standardized voice signal;

[0008] Step 2: Perturb the standardized speech signal to generate a perturbed speech signal;

[0009] Step 3: Input the perturbed speech information into the improved HuBERT model. The improvement of the improved HuBERT model is that it introduces a dynamic semantic masking module and a gated memory fusion module to extract the speech context representation vector.

[0010] Step 4: Input the speech context representation vector into a temporal convolutional attention network, which is composed of alternating layers of convolutional sub-modules and attention sub-modules to extract frame-level temporal context features;

[0011] Step 5: Perform label classification on the frame-level temporal context features based on the DP-means clustering algorithm to obtain the frame-level speech anomaly label sequence;

[0012] Step 6: Input the frame-level speech anomaly label sequence into the structured diagnosis output module, and generate structured diagnosis results by combining the time index;

[0013] Step 7: Send the structured diagnostic results to the voice call platform to achieve linkage between voice diagnosis and call service.

[0014] Optionally, the preprocessing specifically includes:

[0015] The collected speech signals are subjected to silence detection and removal to obtain valid speech segments;

[0016] The effective speech segment is divided into frames according to a set frame length and frame shift, wherein the frame length is 20ms to 30ms and the frame shift is 10ms to 20ms, thereby obtaining frame-level effective speech segments.

[0017] The effective speech segments at the frame level are smoothed and short-time Fourier transform is performed to extract spectral features;

[0018] The spectral features are amplitude normalized to obtain a standardized speech signal.

[0019] Optionally, step two specifically includes:

[0020] The standardized voice signal is copied into a first copy, a second copy, and a third copy, respectively.

[0021] Apply a spectrum blocking perturbation to the first copy to mask the amplitude component corresponding to a preset frequency band;

[0022] Apply a temporal scale perturbation to the second copy and change the frame sequence density through interpolation operations;

[0023] An amplitude perturbation is applied to the third copy, and a preset multiplicative coefficient transformation is applied to each frame of the speech signal to adjust the amplitude;

[0024] The first, second, and third copies of the perturbed audio signal are extracted at the frame level and then randomly shuffled and reordered to generate a complete audio sequence.

[0025] Optionally, the dynamic semantic masking module includes a frame confidence evaluation unit, a mask generation unit, and a filtering and fusion unit;

[0026] The frame confidence evaluation unit is used to evaluate the validity of each frame in the disturbed speech signal and calculate the frame-level confidence score for each frame, specifically:

[0027] The energy value of the current frame is obtained by summing the squares of the amplitudes of all sampling points in the current frame.

[0028] Centered on the current frame, select a set number of neighboring frames forward and backward, and calculate the average energy of the neighboring frames;

[0029] Calculate the absolute value of the difference between the energy value of the current frame and the average energy value, divide it by the average energy value to obtain the ratio, and subtract the ratio from 1 to obtain the energy stability score;

[0030] Calculate the Euclidean distance between the feature vectors of the normalized speech signal and the perturbed speech signal at the same frame position, add 1 to the Euclidean distance and then take the reciprocal to obtain the perturbed fidelity score;

[0031] The energy stability score and the perturbation fidelity score are weighted and fused according to a preset weight to calculate the frame-level confidence score for each frame.

[0032] The mask generation unit is used to filter the frame-level confidence score according to a set score threshold and generate a frame-level mask vector. Specifically, when the frame-level confidence score is less than the set score threshold, the corresponding frame-level mask vector position is assigned a value of 0; when the frame-level confidence score is greater than or equal to the set score threshold, the corresponding frame-level mask vector position is assigned a value of 1.

[0033] The filtering and fusion unit is used to perform frame-by-frame dot product processing on the initial frame vector and the frame-level mask vector output by the improved HuBERT model encoder to obtain the current frame vector.

[0034] The current frame vector is input into the gated memory fusion module, which includes a gated calculation unit and a memory update unit.

[0035] The gating calculation unit is used to receive the frame vectors of the current layer and the previous layer, and calculate the gating weights through the Sigmoid activation function;

[0036] The memory update unit performs weighted fusion of the current layer frame vector and the previous layer frame vector according to the gating weight, and outputs the updated frame vector as the input of the next layer, finally obtaining the speech context representation vector.

[0037] Optionally, step four specifically includes:

[0038] The speech context representation vector is input into the first convolutional submodule, which includes a one-dimensional convolutional layer, a non-linear activation function layer, and a normalization layer, to extract local temporal context features and output the frame-level representation vector after convolution processing.

[0039] The frame-level representation vector after convolution is input into the attention submodule of the corresponding layer. The attention submodule models the global dependencies between frames, specifically including:

[0040] The frame-level representation vector is converted into a query vector, a key vector, and a value vector respectively through linear mapping;

[0041] Calculate the inner product between the query vector of each frame and the key vectors of all frames, and divide by the square root of the vector dimension to obtain the attention score.

[0042] Perform Softmax normalization on the attention scores to obtain the inter-frame attention weight distribution;

[0043] The inter-frame attention weight distribution is weighted and summed with the corresponding value vector to generate an attention representation vector that integrates inter-frame context information, which is used as the output frame-level representation vector of the current layer.

[0044] The output frame-level representation vector of the current layer is used as the input of the next layer's convolutional submodule until all alternately stacked convolutional submodules and attention submodules complete forward propagation, ultimately obtaining frame-level temporal context features.

[0045] Optionally, step five specifically includes:

[0046] The frame-level temporal context features are standardized to unify the distribution range of feature values ​​in each dimension;

[0047] Randomly select a frame-level temporal context feature as the initial cluster center, and set a preset distance threshold as the condition for determining new clusters;

[0048] For each frame-level temporal context feature, calculate the Euclidean distance between it and all existing cluster centers in turn;

[0049] If the minimum Euclidean distance is less than the preset distance threshold, the current frame-level temporal context features are assigned to the corresponding cluster.

[0050] If the minimum Euclidean distance is greater than or equal to the preset distance threshold, then the current frame-level temporal context feature is used as the new cluster center to create a new cluster.

[0051] After completing the initial allocation of all frame-level temporal context features, the arithmetic mean of all frame-level temporal context feature vectors in each cluster is calculated in each dimension, and the corresponding cluster center is updated.

[0052] Repeat the frame-level temporal context feature allocation and cluster center update process until one of the following convergence conditions is met:

[0053] Condition 1: The cluster labels of all frame-level temporal context features remain unchanged during two adjacent rounds of clustering;

[0054] Condition 2: The change in the Euclidean distance between all cluster centers is less than the set convergence threshold;

[0055] After clustering is completed, the cluster number of each frame is recorded in frame order to form a frame-level speech anomaly label sequence.

[0056] Optionally, step six specifically includes:

[0057] The frame-level speech anomaly label sequence is input into the structured diagnostic output module, and the temporal continuity analysis of the frame-level speech anomaly label sequence is performed to identify continuous segments with consistent labels in adjacent frames.

[0058] For each continuous segment, extract the start frame and the end frame, and convert them into the corresponding timestamp range based on the frame shift time interval;

[0059] Construct structured diagnostic entries, including speech anomaly label type, start time, and end time;

[0060] For each structured diagnostic entry, the duration of the corresponding time segment is calculated and compared with a set duration threshold. Structured diagnostic entries with a duration greater than the set duration threshold are selected and arranged in time index order to obtain the structured diagnostic results.

[0061] Optionally, step seven specifically includes:

[0062] The structured diagnostic results are organized into an information format that includes voice anomaly label type, corresponding time range and session identifier. The session identifier is used to uniquely identify a voice interaction process and is sent to the voice call platform.

[0063] The voice call platform receives information format data and takes corresponding service actions based on the voice anomaly label type and corresponding time range, thereby realizing the linkage between voice diagnosis and call service.

[0064] The deep learning-based voice diagnostic call system according to an embodiment of the present invention includes the following modules:

[0065] The voice acquisition module is used to collect the user's voice signals during voice interaction.

[0066] The preprocessing module is used to preprocess the speech signal to generate a standardized speech signal;

[0067] The perturbation enhancement module is used to copy the standardized speech signal into multiple copies and apply spectral occlusion perturbation, time scale perturbation and amplitude perturbation to each copy respectively, and generate a perturbation speech signal by scrambling and rearranging frame-level segments.

[0068] The semantic representation extraction module is used to receive perturbed speech signals and extract speech context representation vectors based on the improved HuBERT model that incorporates a dynamic semantic masking module and a gated memory fusion module.

[0069] A frame-level context modeling module is used to receive the speech context representation vector based on a temporal convolutional attention network, which is composed of multiple layers of convolutional sub-modules and attention sub-modules stacked alternately to extract frame-level temporal context features.

[0070] The label clustering module is used to perform label classification on the frame-level temporal context features based on the DP-means clustering algorithm and output a frame-level speech anomaly label sequence;

[0071] The structured diagnostic output module is used to receive frame-level speech anomaly label sequences, perform temporal continuity analysis and segment filtering, and generate structured diagnostic results by combining frame shift time intervals.

[0072] The result push module is used to encapsulate the structured diagnostic results and corresponding session identifiers into data in a unified format and send them to the voice call platform for the voice call platform to perform call linkage processing.

[0073] According to an embodiment of the present invention, a deep learning-based voice diagnostic call storage medium stores a computer program that, when executed by a processor, enables the processor to execute a deep learning-based voice diagnostic call method.

[0074] The beneficial effects of this invention are:

[0075] This invention constructs perturbation speech signals of various perturbation types through a perturbation enhancement module, improving the model's robustness in perceiving frame-level speech anomalies. Addressing the issue of frame-level instability in perturbation signals, which is difficult for the standard HuBERT model to handle, a dynamic semantic masking module and a gated memory fusion module are introduced into the semantic representation extraction module. This significantly improves the HuBERT model's capabilities in frame-level feature filtering and upper / lower layer memory interaction. The dynamic semantic masking module effectively removes invalid frames introduced during the perturbation enhancement process through a frame confidence evaluation unit and a mask generation unit. The gated memory fusion module uses a gating mechanism to adjust the fusion weights of the current and previous layer frame vectors, enhancing the stability and semantic coherence of the speech context representation vector. The frame-level context modeling module employs a temporal convolutional attention network, using an alternating stacked structure of multi-layer convolutional sub-modules and attention sub-modules to balance local temporal context modeling and long-distance dependency modeling, capturing the internal temporal features of the speech.

[0076] In the label clustering module, this invention introduces an unsupervised DP-means clustering algorithm, which automatically determines the number of clusters without relying on prior assumptions about the number of categories. Cluster centers are dynamically created by setting a preset distance threshold, and the cluster center mean update and Euclidean distance convergence criteria are introduced to ensure the clustering accuracy of frame-level temporal context features. The final generated frame-level speech anomaly label sequence has a clear time index structure. After being input into the structured diagnostic output module, combined with the frame shift time interval, it can accurately identify the time range of label segments. Short-term unstable labels are filtered out using an effective duration threshold strategy. Finally, the result push module links with the voice call platform to achieve automated call service response driven by diagnostic results, improving the intelligence and service efficiency of the voice interaction system. Attached Figure Description

[0077] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0078] Figure 1 This is an overall flowchart of the deep learning-based voice diagnostic call method proposed in this invention;

[0079] Figure 2 This is a schematic diagram of the structure of the deep learning-based voice diagnostic call system proposed in this invention. Detailed Implementation

[0080] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0081] refer to Figure 1The deep learning-based voice diagnostic call method includes the following steps:

[0082] Step 1: Collect the user's voice signal during the call interaction process, and preprocess the voice signal to obtain a standardized voice signal;

[0083] Step 2: Perturb the standardized speech signal to generate a perturbed speech signal;

[0084] Step 3: Input the perturbed speech information into the improved HuBERT model. The improvement of the improved HuBERT model is that it introduces a dynamic semantic masking module and a gated memory fusion module to extract the speech context representation vector.

[0085] Step 4: Input the speech context representation vector into a temporal convolutional attention network, which is composed of alternating layers of convolutional sub-modules and attention sub-modules to extract frame-level temporal context features;

[0086] Step 5: Perform label classification on the frame-level temporal context features based on the DP-means clustering algorithm to obtain the frame-level speech anomaly label sequence;

[0087] Step 6: Input the frame-level speech anomaly label sequence into the structured diagnosis output module, and generate structured diagnosis results by combining the time index;

[0088] Step 7: Send the structured diagnostic results to the voice call platform to achieve linkage between voice diagnosis and call service.

[0089] In this embodiment, the preprocessing specifically includes:

[0090] The collected speech signals are subjected to silence detection and removal to obtain valid speech segments;

[0091] The effective speech segment is divided into frames according to a set frame length and frame shift, wherein the frame length is 20ms to 30ms and the frame shift is 10ms to 20ms, thereby obtaining frame-level effective speech segments.

[0092] The effective speech segments at the frame level are smoothed and short-time Fourier transform is performed to extract spectral features;

[0093] The spectral features are amplitude normalized to obtain a standardized speech signal.

[0094] In this embodiment, step two specifically includes:

[0095] The standardized voice signal is copied into a first copy, a second copy, and a third copy, respectively.

[0096] Apply a spectrum blocking perturbation to the first copy to mask the amplitude component corresponding to a preset frequency band;

[0097] Apply a temporal scale perturbation to the second copy and change the frame sequence density through interpolation operations;

[0098] An amplitude perturbation is applied to the third copy, and a preset multiplicative coefficient transformation is applied to each frame of the speech signal to adjust the amplitude;

[0099] The first, second, and third copies of the perturbation are extracted into speech segments at the frame level, and a complete speech sequence is generated by randomly shuffling and reordering them to form a perturbation speech signal.

[0100] In the process of applying spectrum blocking perturbation to the first copy, this invention selects a set of frequency band index ranges as blocking areas in each frame of spectrum map according to a preset blocking strategy. The selected frequency band areas can be continuous or discontinuous segments. During the blocking process, the amplitude component corresponding to the target frequency band is zeroed out, or the amplitude value of the mirror frame, neighboring frame, or adjacent frequency band is replaced, thereby simulating the situation of missing or damaged channel frequency domain information and constructing perturbation data with spectrum missing characteristics.

[0101] During the application of temporal perturbation to the second copy, interpolation is performed on the original frame sequence based on the timestamp of each frame and the inter-frame relationships to change the temporal distribution density of the frames. Specifically, spline interpolation is used to insert new synthetic frames between adjacent frames, or to remove some frames from the frame sequence, thereby compressing and stretching the speech signal on a temporal scale. After interpolation, the temporal waveform or spectral representation is reconstructed to possess dynamic speech rate variation characteristics, thereby enhancing the model's ability to recognize speech under different speech rate conditions.

[0102] During the process of applying amplitude perturbation to the third copy, for each frame of the speech signal, its amplitude information is extracted and a preset multiplicative coefficient is applied. The multiplicative coefficient is a normally distributed random variable, which is applied to the original amplitude signal in the form of frame-by-frame multiplication to form an amplitude compression or enhancement effect.

[0103] In this embodiment, the dynamic semantic masking module includes a frame confidence evaluation unit, a mask generation unit, and a filtering and fusion unit;

[0104] The frame confidence evaluation unit is used to evaluate the validity of each frame in the disturbed speech signal and calculate the frame-level confidence score for each frame, specifically:

[0105] The energy value of the current frame is obtained by summing the squares of the amplitudes of all sampling points in the current frame.

[0106] Centered on the current frame, select a set number of neighboring frames forward and backward, and calculate the average energy of the neighboring frames;

[0107] Calculate the absolute value of the difference between the energy value of the current frame and the average energy value, divide it by the average energy value to obtain the ratio, and subtract the ratio from 1 to obtain the energy stability score;

[0108]

[0109] Among them, S e (t) represents the energy stability score of the t-th frame, E t This represents the energy value of frame t. This represents the average energy of the neighboring frames of frame t.

[0110] The Euclidean distance between the feature vectors of the standardized speech signal and the perturbed speech signal at the same frame position is calculated. The Euclidean distance is incremented by 1 and then the reciprocal is taken to obtain the perturbed fidelity score. The frame-level feature vectors of the standardized speech signal and the perturbed speech signal are obtained by performing short-time Fourier transform on their respective speech frames. The amplitude spectrum of each frame is taken as the feature vector of the corresponding frame.

[0111]

[0112] Among them, S d (t) represents the perturbation fidelity score of the t-th frame, x t Let y represent the feature vector of the normalized speech signal in frame t. t Let represent the feature vector of the perturbed speech signal in frame t, and ||·||2 represent the L2 norm;

[0113] The energy stability score and the perturbation fidelity score are weighted and fused according to a preset weight to calculate the frame-level confidence score for each frame.

[0114] S(t)=α·S e (t)+(1-α)·S d (t);

[0115] Where S(t) represents the frame-level confidence score of the t-th frame, and α represents the preset weight;

[0116] The mask generation unit is used to filter the frame-level confidence score according to a set score threshold and generate a frame-level mask vector. Specifically, when the frame-level confidence score is less than the set score threshold, the corresponding frame-level mask vector position is assigned a value of 0; when the frame-level confidence score is greater than or equal to the set score threshold, the corresponding frame-level mask vector position is assigned a value of 1.

[0117] The filtering and fusion unit is used to perform frame-by-frame dot product processing on the initial frame vector and the frame-level mask vector output by the improved HuBERT model encoder to obtain the current frame vector.

[0118] The current frame vector is input into the gated memory fusion module, which includes a gated calculation unit and a memory update unit.

[0119] The gating calculation unit is used to receive the frame vectors of the current layer and the previous layer, and calculate the gating weights using the Sigmoid activation function:

[0120]

[0121] Among them, g t This represents the gating weight for frame t. This represents the vector output of the l-th layer at frame t. denoted as the vector output of the (l-1)th layer at frame t, where Sigmoid represents the activation function, W1 and W2 represent the weight matrices, and b represents the bias vector;

[0122] The memory update unit performs weighted fusion of the current layer frame vector and the previous layer frame vector according to the gating weight, and outputs the updated frame vector, which is used as the input of the next layer, and finally obtains the speech context representation vector.

[0123]

[0124] in, This represents the updated frame vector at layer l;

[0125] In existing technologies, the HuBERT (Hidden-Unit BERT) model is a self-supervised speech pre-training model based on the Transformer architecture. Its basic structure consists of two main parts: a front-end convolutional encoder and a context modeling module composed of multiple stacked Transformers. The original HuBERT model mainly learns high-level speech representations by converting the raw speech signal into frame-level feature input to the encoder, and then using a self-supervised learning approach guided by discrete labels for context modeling. However, this structure has certain limitations in handling complex speech scenarios (such as dialect variations, noise perturbations, and semantic loss in call dialogues) in terms of identifying and preserving information from anomalous frames, especially lacking fine-grained control mechanisms for frame-level quality control and semantic representation continuity.

[0126] To address this, this invention structurally improves the HuBERT model by introducing a dynamic semantic masking module and a gated memory fusion module, embedded between the encoder output layer and the Transformer stacking layers, respectively. The dynamic semantic masking module, deployed between the convolutional encoder and the Transformer input, performs frame-level confidence evaluation on the perturbation-enhanced speech frame features, generates a dynamic mask vector, and removes low-confidence frame features, thereby improving the input purity in the semantic encoding stage. This module comprises three substructures: a frame confidence evaluation unit, a mask generation unit, and a filtering fusion unit, forming a complete dynamic filtering path.

[0127] Furthermore, a gated memory fusion module is embedded between each Transformer layer to control the memory of frame features at each layer. This module calculates gate weights based on the similarity of frame vectors between the previous and current layers, and preserves long-term dependency information between upper and lower layers through linear fusion, mitigating the semantic drift problem in deep networks. By retaining information layer by layer with weights, the gated memory module enables the model to have stronger multi-layer semantic aggregation capabilities, effectively enhancing the deep expressive power of context modeling.

[0128] In this embodiment, step four specifically includes:

[0129] The speech context representation vector is input into the first convolutional submodule, which includes a one-dimensional convolutional layer, a non-linear activation function layer, and a normalization layer, to extract local temporal context features and output the frame-level representation vector after convolution processing.

[0130] The frame-level representation vector after convolution is input into the attention submodule of the corresponding layer. The attention submodule models the global dependencies between frames, specifically including:

[0131] The frame-level representation vector is converted into a query vector, a key vector, and a value vector respectively through linear mapping;

[0132] Calculate the inner product between the query vector of each frame and the key vectors of all frames, and divide by the square root of the vector dimension to obtain the attention score.

[0133] Perform Softmax normalization on the attention scores to obtain the inter-frame attention weight distribution;

[0134] The inter-frame attention weight distribution is weighted and summed with the corresponding value vector to generate an attention representation vector that integrates inter-frame context information, which is used as the output frame-level representation vector of the current layer.

[0135] The output frame-level representation vector of the current layer is used as the input of the next layer's convolutional submodule until all the alternately stacked convolutional submodules and attention submodules have completed forward propagation, and finally the frame-level temporal context features are obtained.

[0136] In its implementation, the temporal convolutional attention network achieves layer-by-layer deep modeling of the speech context representation vector through alternating stacking of convolutional and attention sub-modules. The convolutional sub-modules primarily focus on the contextual patterns of local regions within a frame sequence, extracting feature correlations between temporally adjacent frames through one-dimensional convolution operations, enhancing non-linear expressive power by combining non-linear activation functions, and stabilizing the training process through normalization operations. The attention sub-modules receive the output of the convolutional sub-modules, concatenate all frame-level representation vectors into a sequence input, and map it to three sets of vectors—query, key, and value—through linear transformations to capture global inter-frame dependencies. In the attention mechanism, the inner product between the query vector of any frame and the key vectors of all frames is calculated and divided by the square root of the vector dimension to obtain the attention score. After Softmax normalization, a weight distribution is generated, which is used to weight and aggregate the value vectors to form a frame representation of global semantic fusion. This process is repeated with multiple layers of convolution-attention combination, enabling the network to simultaneously model short-term dynamics and long-term dependencies. The final output frame-level temporal context features can effectively characterize the global dynamic structural characteristics of speech signals, providing a stable and high-quality input foundation for subsequent frame-level classification and structured diagnosis.

[0137] In this embodiment, step five specifically includes:

[0138] The frame-level temporal context features are standardized to unify the distribution range of feature values ​​in each dimension;

[0139] Randomly select a frame-level temporal context feature as the initial cluster center, and set a preset distance threshold as the condition for determining new clusters;

[0140] For each frame-level temporal context feature, calculate the Euclidean distance between it and all existing cluster centers in turn;

[0141] If the minimum Euclidean distance is less than the preset distance threshold, the current frame-level temporal context features are assigned to the corresponding cluster.

[0142] If the minimum Euclidean distance is greater than or equal to the preset distance threshold, then the current frame-level temporal context feature is used as the new cluster center to create a new cluster.

[0143] After completing the initial allocation of all frame-level temporal context features, the arithmetic mean of all frame-level temporal context feature vectors in each cluster is calculated in each dimension, and the corresponding cluster center is updated.

[0144] Repeat the frame-level temporal context feature allocation and cluster center update process until one of the following convergence conditions is met:

[0145] Condition 1: The cluster labels of all frame-level temporal context features remain unchanged during two adjacent rounds of clustering;

[0146] Condition 2: The change in the Euclidean distance between all cluster centers is less than the set convergence threshold;

[0147] After clustering is completed, the cluster number of each frame is recorded in frame order to form a frame-level speech anomaly label sequence.

[0148] In this embodiment, step six specifically includes:

[0149] The frame-level speech anomaly label sequence is input into the structured diagnostic output module, and the temporal continuity analysis of the frame-level speech anomaly label sequence is performed to identify continuous segments with consistent labels in adjacent frames.

[0150] For each continuous segment, extract the start frame and the end frame, and convert them into the corresponding timestamp range based on the frame shift time interval;

[0151] Construct structured diagnostic entries, including speech anomaly label type, start time, and end time;

[0152] For each structured diagnostic entry, the duration of the corresponding time segment is calculated and compared with a set duration threshold. Structured diagnostic entries with a duration greater than the set duration threshold are selected and arranged in time index order to obtain the structured diagnostic results.

[0153] In this embodiment, step seven specifically includes:

[0154] The structured diagnostic results are organized into an information format that includes voice anomaly label type, corresponding time range and session identifier. The session identifier is used to uniquely identify a voice interaction process and is sent to the voice call platform.

[0155] The voice call platform receives information format data and takes corresponding service actions based on the voice anomaly label type and corresponding time range, thereby realizing the linkage between voice diagnosis and call service.

[0156] In practical applications, the voice call platform can be deployed in intelligent customer service systems to assist in completing voice screening tasks related to user health.

[0157] refer to Figure 2 A deep learning-based voice diagnostic call system includes the following modules:

[0158] The voice acquisition module is used to collect the user's voice signals during voice interaction.

[0159] The preprocessing module is used to preprocess the speech signal to generate a standardized speech signal;

[0160] The perturbation enhancement module is used to copy the standardized speech signal into multiple copies and apply spectral occlusion perturbation, time scale perturbation and amplitude perturbation to each copy respectively, and generate a perturbation speech signal by scrambling and rearranging frame-level segments.

[0161] The semantic representation extraction module is used to receive perturbed speech signals and extract speech context representation vectors based on the improved HuBERT model that incorporates a dynamic semantic masking module and a gated memory fusion module.

[0162] A frame-level context modeling module is used to receive the speech context representation vector based on a temporal convolutional attention network, which is composed of multiple layers of convolutional sub-modules and attention sub-modules stacked alternately to extract frame-level temporal context features.

[0163] The label clustering module is used to perform label classification on the frame-level temporal context features based on the DP-means clustering algorithm and output a frame-level speech anomaly label sequence;

[0164] The structured diagnostic output module is used to receive frame-level speech anomaly label sequences, perform temporal continuity analysis and segment filtering, and generate structured diagnostic results by combining frame shift time intervals.

[0165] The result push module is used to encapsulate the structured diagnostic results and corresponding session identifiers into data in a unified format and send them to the voice call platform for the voice call platform to perform call linkage processing.

[0166] A deep learning-based voice diagnostic call storage medium stores a computer program that, when executed by a processor, enables the processor to perform a deep learning-based voice diagnostic call method.

[0167] Example 1:

[0168] To verify the feasibility of this invention in practice, it was applied to a call analysis system in a company's voice customer service center. This system is used to perform real-time voice analysis on call data from users in after-sales, consultation, and complaint scenarios to determine whether there are abnormal acoustic manifestations such as abnormal breathing, emotional tension, and language barriers. It also automatically prompts human agents to pay attention through a linked process, thereby optimizing service response efficiency and user experience.

[0169] During project deployment, the system first collects the audio stream from the user's end during a call using the voice acquisition module. Then, the preprocessing module performs silence removal, framing, Fourier transform, and normalization to standardize the audio of varying quality. The perturbation enhancement module generates multiple perturbation versions of the standardized audio signal, creating training samples that incorporate spectral occlusion, temporal scale variations, and amplitude perturbations. This enhances the subsequent model's adaptability to different call environments.

[0170] In the core semantic feature extraction stage, the system adopts an improved HuBERT model that incorporates a dynamic semantic masking module and a gated memory fusion module. This automatically ignores low-confidence frames while maintaining contextual consistency between high-value frames, making the output speech context representation vectors more stable and semantically continuous. A temporal convolutional attention network is then used to model the inter-frame dependencies of these representation vectors, taking into account both short-term local features and long-term dependencies, thus comprehensively extracting frame-level temporal contextual features.

[0171] In the clustering stage, the system applies the DP-means clustering algorithm to classify and label frame-level temporal context features. Compared to the traditional K-means algorithm, DP-means does not require a preset number of clusters; instead, it dynamically generates cluster centers based on a set distance threshold, demonstrating stronger adaptability and label separation in real-world scenarios. The final output frame-level speech anomaly label sequence is then processed by the structured diagnostic output module to generate time-localized label segments, which are then pushed to the speech platform to trigger subsequent strategy processing.

[0172] Table 1. Performance Comparison of the Invention System and the Control System in Anomaly Detection Tasks

[0173]

[0174] Analysis of the experimental results in Table 1 above clearly shows that the overall performance of the system of the present invention in the anomaly identification task is superior to that of the control system.

[0175] The existing system uses a traditional LSTM network for feature modeling and combines it with fixed symptom labeling rules for recognition. While it has some temporal modeling capabilities, it performs poorly in handling complex speech information and dynamic symptom changes. In contrast, this invention uses an improved HuBERT model for semantic modeling, introducing a dynamic semantic masking module and a gated memory fusion module to effectively enhance the expressive power of frame-level features. Furthermore, it employs the DP-means algorithm in the clustering stage to achieve adaptive generation of frame-level labels, avoiding the risk of misidentification that may arise from fixed rules.

[0176] Performance data shows that the system of this invention achieves a frame-level anomaly recognition accuracy of 93.4%, a significant improvement over the control system's 78.5%, indicating that the system can more accurately identify potential anomalies in speech. The average time deviation for anomaly segment localization is ±0.38 seconds, far superior to the control system's ±1.12 seconds, demonstrating its advantage in temporal localization accuracy. The shorter average call response time (2.6 seconds vs. 6.8 seconds) further illustrates the system's faster response speed, which is beneficial for meeting real-time requirements in practical deployments.

[0177] Regarding the false alarm rate, the system of this invention has a false alarm rate of only 2.1%, while the control system has a false alarm rate of 8.7%, significantly reducing false identification and demonstrating good robustness. The recall rate of abnormal segments is 91.0%, higher than the 72.6% of the control system. Recall rate is an indicator of a system's ability to identify and cover all real abnormal segments; a high recall rate indicates that the system effectively controls the risk of missing abnormalities. The label F1 score is 0.915, significantly higher than the 0.728 of the control system. The F1 score balances precision and recall and is an important indicator of overall recognition capability, demonstrating that the system of this invention has significant advantages in both stability and accuracy.

[0178] To further verify the robustness and stability of the system under various voice input environments, six typical test scenarios were constructed. In each scenario, 100 test voice segments were input, and the system was used to identify and locate voice anomaly labels. The consistency of the system output results with the manually labeled results and the localization error were evaluated. The test results are shown in Table 2 below.

[0179] Table 2. Consistency of frame-level tag output of the system of the present invention under various speech disturbance environments.

[0180] Test conditions Label consistency rate (%) Average deviation (seconds) Standard pronunciation 97.8 0.21 Analog telephone compression (8kHz) 94.6 0.30 Background noise interference (SNR 10dB) 93.2 0.34 Speech speed increased by 30% 91.9 0.36 Simulates a Mandarin accent (mixed with dialects). 90.5 0.38 There are many silent or intermittent segments. 91.2 0.37

[0181] Table 2 above shows the consistency evaluation results of the frame-level tag output of the system of the present invention under various speech disturbance environments. The system's robustness and positioning accuracy under different test conditions are reflected from two dimensions: tag consistency rate and average deviation.

[0182] Under standard speech conditions, the system of this invention performs best, with a tag consistency rate of 97.8% and an average deviation of only 0.21 seconds. This indicates that in a undisturbed environment, the system can accurately identify frame-level speech anomaly tags and achieve precise positioning.

[0183] Under analog telephone compression (8kHz) conditions, the label consistency rate still reached 94.6%, with an average deviation of 0.30 seconds, indicating that even with limited audio segments and compression distortion, the system still has good adaptability. This is due to the improved HuBERT expressive ability of this invention in preserving compressed speech features.

[0184] Under background noise interference (SNR 10dB), the label consistency rate of the system of this invention is 93.2%, with an average deviation of 0.34 seconds. Although noise poses a challenge to speech feature extraction, this system effectively suppresses noise interference through frame-level attention mechanism and multi-layer modeling strategy, maintaining high label output stability.

[0185] Even with a 30% increase in speech rate, the label consistency rate was 91.9%, with an average deviation of 0.36 seconds. While changes in speech rate affect the rhythm and prosodic information of the speech signal, the system of this invention captures sufficient local temporal features through a temporal convolutional network, ensuring parsing accuracy.

[0186] In a simulated Mandarin accent (dialect mix) scenario, the label consistency rate was 90.5%, the lowest among all scenarios, but still at a relatively high level, with an average deviation controlled at 0.38 seconds. This indicates that while the system has some sensitivity to dialect confusion scenarios, its overall recognition performance remains acceptable.

[0187] In scenarios with numerous silences and intermittent segments, the label consistency rate was 91.2%, with an average deviation of 0.37 seconds. Intermittent signals often lead to sparse or discontinuous frame-level feature information. However, by using the DP-means clustering algorithm for adaptive clustering of feature distribution, the system of this invention can effectively preserve structured abnormal features and avoid silence errors.

[0188] This embodiment fully verifies the practicality and superior performance of the invention in complex call environments by applying it to a real-world voice customer service scenario. The system introduces an improved HuBERT model into the voice processing flow, incorporating a dynamic semantic mask module and a gated memory fusion module. This makes frame-level voice feature representation more stable and accurate, significantly improving the ability to identify abnormal voice performance. In the anomaly classification stage, the DP-means clustering algorithm replaces the traditional fixed rules and K-means clustering method, achieving adaptive clustering and labeling of frame-level features, effectively reducing the risk of misidentification caused by rigid rules or unreasonable cluster number settings. The structured output module, combined with time indexing, accurately labels abnormal segments and automatically links with the voice platform to quickly transmit abnormal information and trigger strategies, significantly optimizing the timeliness and accuracy of service responses. Overall, the system of this invention demonstrates significant advantages in accuracy, stability, and practicality, effectively solving the problems of poor real-time performance, inaccurate localization, and high false alarm rates in traditional voice recognition systems for anomaly detection. It possesses good engineering adaptability and promising prospects for widespread application.

[0189] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A voice diagnostic call method based on deep learning, characterized in that, Includes the following steps: Step 1: Collect the user's voice signal during the call interaction process, and preprocess the voice signal to obtain a standardized voice signal; Step 2: Perturb the standardized speech signal to generate a perturbed speech signal; Step 3: Input the perturbed speech information into the improved HuBERT model. The improvement of the improved HuBERT model is that it introduces a dynamic semantic masking module and a gated memory fusion module to extract the speech context representation vector. Step 4: Input the speech context representation vector into a temporal convolutional attention network, which is composed of alternating layers of convolutional sub-modules and attention sub-modules to extract frame-level temporal context features; Step 5: Perform label classification on the frame-level temporal context features based on the DP-means clustering algorithm to obtain the frame-level speech anomaly label sequence; Step 6: Input the frame-level speech anomaly label sequence into the structured diagnosis output module, and generate structured diagnosis results by combining the time index; Step 7: Send the structured diagnostic results to the voice call platform to achieve linkage between voice diagnosis and call service; The dynamic semantic masking module includes a frame confidence evaluation unit, a mask generation unit, and a filtering and fusion unit; The frame confidence evaluation unit is used to evaluate the validity of each frame in the disturbed speech signal and calculate the frame-level confidence score for each frame, specifically: The energy value of the current frame is obtained by summing the squares of the amplitudes of all sampling points in the current frame. Centered on the current frame, select a set number of neighboring frames forward and backward, and calculate the average energy of the neighboring frames; Calculate the absolute value of the difference between the energy value of the current frame and the average energy value, divide it by the average energy value to obtain the ratio, and subtract the ratio from 1 to obtain the energy stability score; Calculate the Euclidean distance between the feature vectors of the normalized speech signal and the perturbed speech signal at the same frame position, add 1 to the Euclidean distance and then take the reciprocal to obtain the perturbed fidelity score; The energy stability score and the perturbation fidelity score are weighted and fused according to a preset weight to calculate the frame-level confidence score for each frame. The mask generation unit is used to filter the frame-level confidence score according to a set score threshold and generate a frame-level mask vector. Specifically, when the frame-level confidence score is less than the set score threshold, the corresponding frame-level mask vector position is assigned a value of 0; when the frame-level confidence score is greater than or equal to the set score threshold, the corresponding frame-level mask vector position is assigned a value of 1. The filtering and fusion unit is used to perform frame-by-frame dot product processing on the initial frame vector and the frame-level mask vector output by the improved HuBERT model encoder to obtain the current frame vector. The current frame vector is input into the gated memory fusion module, which includes a gated calculation unit and a memory update unit. The gating calculation unit is used to receive the frame vectors of the current layer and the previous layer, and calculate the gating weights through the Sigmoid activation function; The memory update unit performs weighted fusion of the current layer frame vector and the previous layer frame vector according to the gating weight, and outputs the updated frame vector as the input of the next layer, finally obtaining the speech context representation vector.

2. The deep learning-based voice diagnostic call method according to claim 1, characterized in that, The preprocessing specifically includes: The collected speech signals are subjected to silence detection and removal to obtain valid speech segments; The effective speech segment is divided into frames according to a set frame length and frame shift, wherein the frame length is 20ms to 30ms and the frame shift is 10ms to 20ms, thereby obtaining frame-level effective speech segments. The effective speech segments at the frame level are smoothed and short-time Fourier transform is performed to extract spectral features; The spectral features are amplitude normalized to obtain a standardized speech signal.

3. The deep learning-based voice diagnostic call method according to claim 1, characterized in that, Step two specifically involves: The standardized voice signal is copied into a first copy, a second copy, and a third copy, respectively. Apply a spectrum blocking perturbation to the first copy to mask the amplitude component corresponding to a preset frequency band; Apply a temporal scale perturbation to the second copy and change the frame sequence density through interpolation operations; An amplitude perturbation is applied to the third copy, and a preset multiplicative coefficient transformation is applied to each frame of the speech signal to adjust the amplitude; The first, second, and third copies of the perturbed audio signal are extracted at the frame level and then randomly shuffled and reordered to generate a complete audio sequence.

4. The deep learning-based voice diagnostic call method according to claim 1, characterized in that, Step four specifically involves: The speech context representation vector is input into the first convolutional submodule, which includes a one-dimensional convolutional layer, a non-linear activation function layer, and a normalization layer, to extract local temporal context features and output the frame-level representation vector after convolution processing. The frame-level representation vector after convolution is input into the attention submodule of the corresponding layer. The attention submodule models the global dependencies between frames, specifically including: The frame-level representation vector is converted into a query vector, a key vector, and a value vector respectively through linear mapping; Calculate the inner product between the query vector of each frame and the key vectors of all frames, and divide by the square root of the vector dimension to obtain the attention score. Perform Softmax normalization on the attention scores to obtain the inter-frame attention weight distribution; The inter-frame attention weight distribution is weighted and summed with the corresponding value vector to generate an attention representation vector that integrates inter-frame context information, which is used as the output frame-level representation vector of the current layer. The output frame-level representation vector of the current layer is used as the input of the next layer's convolutional submodule until all alternately stacked convolutional submodules and attention submodules complete forward propagation, ultimately obtaining frame-level temporal context features.

5. The deep learning-based voice diagnostic call method according to claim 1, characterized in that, Step five specifically involves: The frame-level temporal context features are standardized to unify the distribution range of feature values ​​in each dimension; Randomly select a frame-level temporal context feature as the initial cluster center, and set a preset distance threshold as the condition for determining new clusters; For each frame-level temporal context feature, calculate the Euclidean distance between it and all existing cluster centers in turn; If the minimum Euclidean distance is less than the preset distance threshold, the current frame-level temporal context features are assigned to the corresponding cluster. If the minimum Euclidean distance is greater than or equal to the preset distance threshold, then the current frame-level temporal context feature is used as the new cluster center to create a new cluster. After completing the initial allocation of all frame-level temporal context features, the arithmetic mean of all frame-level temporal context feature vectors in each cluster is calculated in each dimension, and the corresponding cluster center is updated. Repeat the frame-level temporal context feature allocation and cluster center update process until one of the following convergence conditions is met: Condition 1: The cluster labels of all frame-level temporal context features remain unchanged during two adjacent rounds of clustering; Condition 2: The change in the Euclidean distance between all cluster centers is less than the set convergence threshold; After clustering is completed, the cluster number of each frame is recorded in frame order to form a frame-level speech anomaly label sequence.

6. The deep learning-based voice diagnostic call method according to claim 1, characterized in that, Step six specifically involves: The frame-level speech anomaly label sequence is input into the structured diagnostic output module, and the temporal continuity analysis of the frame-level speech anomaly label sequence is performed to identify continuous segments with consistent labels in adjacent frames. For each continuous segment, extract the start frame and the end frame, and convert them into the corresponding timestamp range based on the frame shift time interval; Construct structured diagnostic entries, including speech anomaly label type, start time, and end time; For each structured diagnostic entry, the duration of the corresponding time segment is calculated and compared with a set duration threshold. Structured diagnostic entries with a duration greater than the set duration threshold are selected and arranged in time index order to obtain the structured diagnostic results.

7. The deep learning-based voice diagnostic call method according to claim 1, characterized in that, Step seven specifically involves: The structured diagnostic results are organized into an information format that includes voice anomaly label type, corresponding time range and session identifier. The session identifier is used to uniquely identify a voice interaction process and is sent to the voice call platform. The voice call platform receives information format data and enables the linkage between voice diagnosis and call services.

8. A deep learning-based voice diagnostic call system, executing the deep learning-based voice diagnostic call method according to any one of claims 1 to 7, characterized in that, Includes the following modules: The voice acquisition module is used to collect the user's voice signals during voice interaction. The preprocessing module is used to preprocess the speech signal to generate a standardized speech signal; The perturbation enhancement module is used to copy the standardized speech signal into multiple copies and apply spectral occlusion perturbation, time scale perturbation and amplitude perturbation to each copy respectively, and generate a perturbation speech signal by scrambling and rearranging frame-level segments. The semantic representation extraction module is used to receive perturbed speech signals and extract speech context representation vectors based on the improved HuBERT model that incorporates a dynamic semantic masking module and a gated memory fusion module. A frame-level context modeling module is used to receive the speech context representation vector based on a temporal convolutional attention network, which is composed of multiple layers of convolutional sub-modules and attention sub-modules stacked alternately to extract frame-level temporal context features. The label clustering module is used to perform label classification on the frame-level temporal context features based on the DP-means clustering algorithm and output a frame-level speech anomaly label sequence; The structured diagnostic output module is used to receive frame-level speech anomaly label sequences, perform temporal continuity analysis and segment filtering, and generate structured diagnostic results by combining frame shift time intervals. The result push module is used to encapsulate the structured diagnostic results and corresponding session identifiers into data in a unified format and send them to the voice call platform for the voice call platform to perform call linkage processing.

9. A deep learning-based voice diagnostic call storage medium, wherein the storage medium stores a computer program that, when executed by a processor, enables the processor to perform the deep learning-based voice diagnostic call method according to any one of claims 1 to 7.

Images