Call data intelligent analysis processing method and system based on voice recognition

Abstract

This invention relates to the field of data analysis and processing technology, specifically including a method and system for intelligent analysis and processing of call data based on speech recognition. The method includes: performing multi-head self-attention mechanism encoding learning on standardized semantic feature vectors and standardized interruption feature vectors; inputting the encoding learning results into a decoder connected to the encoder; and performing feedback processing on the response parameters of the call voice data based on multiple call processing duration parameters. This addresses the technical problem of failing to effectively capture long-distance dependencies and multi-granularity rhythmic changes in interactions, leading to inaccurate decoded call processing duration parameters and difficulty in effectively controlling response parameters. By capturing long-distance dependencies and multi-granularity rhythmic changes in interactions, combined with a gated attention mechanism, and using a Transformer encoder-decoder structure to accurately decode multiple call processing duration parameters, the method dynamically adjusts the response parameters, achieving intelligent closed-loop feedback control of the call service process.

Description

Technical Field

[0001] This invention relates to the field of data analysis and processing technology, specifically to a method and system for intelligent analysis and processing of call data based on speech recognition. Background Technology

[0002] In scenarios such as customer service, emergency dispatch, and intelligent agents, call centers generate massive amounts of voice interaction data every day. Efficiently extracting behavioral characteristics such as response delay, waiting time, and conversation rhythm from these unstructured voices and converting them into quantifiable processing time parameters has become a core requirement for improving service efficiency, optimizing resource allocation, and enhancing user experience.

[0003] However, existing call analysis focuses on keyword recognition or sentiment classification, lacking refined modeling of sentence temporal structure and interruption behavior. Even when temporal analysis is introduced, speech-to-text is often used as the only input. In addition, it ignores non-semantic features that imply interaction intent, such as silence intervals and turn-switching rhythms, making it difficult to effectively capture long-distance dependencies and multi-granular rhythm changes. This results in weak generalization ability of predicted call processing parameters, which cannot support refined operational decisions.

[0004] In summary, existing technologies suffer from the following technical problems: they ignore non-semantic features that imply interaction intent in call analysis, fail to effectively capture long-distance dependencies and multi-granular rhythmic changes in interactions, resulting in inaccurate call processing duration parameters after decoding and making it difficult to implement effective feedback control of response parameters. Summary of the Invention

[0005] This application provides a method and system for intelligent analysis and processing of call data based on speech recognition. It aims to solve the technical problems in existing call analysis that ignore non-semantic features that imply interaction intent, cannot effectively capture long-distance dependencies and multi-granular rhythm changes in the interaction, resulting in inaccurate call processing duration parameters after decoding, and making it difficult to implement effective feedback control of response parameters.

[0006] In view of the above problems, the technical solution to achieve the present application is as follows: In a first aspect, this application provides a method for intelligent analysis and processing of call data based on speech recognition. The method includes: collecting call voice data; performing sentence recognition and timestamp annotation on the call voice data to obtain a sentence time-series mapping sequence; calculating the interruption duration of the sentence time-series mapping sequence and storing it as a standardized sentence time-series mapping sequence; extracting a semantic feature standardized vector and an interruption feature standardized vector from the standardized sentence time-series mapping sequence; in a pre-trained Transformer model, the multi-layer encoder of the Transformer model performs multi-head self-attention mechanism encoding learning on the semantic feature standardized vector and the interruption feature standardized vector; inputting the encoding learning result into a decoder connected to the encoder; the decoder is used to decode and obtain multiple call processing duration parameters; and performing feedback processing on the response parameters of the call voice data based on the multiple call processing duration parameters.

[0007] In a possible implementation, each layer of the multi-layer encoder includes a multi-head self-attention mechanism, a feedforward network layer, and a residual network layer. The multi-head self-attention mechanism includes multiple attention heads for learning user voice cutoff response duration features, call data processing wait duration features, and call feedback rhythm change duration features. The semantic feature normalization vector and the interruption feature normalization vector are concatenated to obtain a concatenated feature normalization vector. The multi-head self-attention mechanism is then used to encode and learn the concatenated feature normalization vector.

[0008] In a possible implementation, each attention head of the multi-head self-attention mechanism is used to calculate a linear mapping matrix based on the concatenated feature normalization vector. The linear mapping matrix includes a key vector feature matrix, a value vector feature matrix, and a query vector feature matrix. Attention weights are calculated on the linear mapping matrix to obtain an attention weight matrix. The weighted output features of the multiple attention weight matrices are concatenated by multiple heads to obtain a multi-head self-attention concatenated output sequence. The multi-head self-attention concatenated output sequence is processed according to the feedforward network layer and the residual network layer to obtain a multi-head self-attention encoded output sequence. The multi-head self-attention encoded output sequence is output as the encoding learning result.

[0009] In a possible implementation, the multi-head self-attention encoding output sequence is sent to the decoder, which includes multiple prediction heads corresponding to the multi-head self-attention mechanism for predicting the user's voice cutoff response duration, the learning call data processing wait duration, and the call feedback rhythm change duration; the encoder when the mean square error of the multiple prediction heads is less than a preset threshold is set to a convergent state, thereby obtaining multiple call processing duration parameters decoded by the decoder when it is in a convergent state.

[0010] In a possible implementation, semantic scene recognition is performed on the call voice data to obtain multiple semantic scenes; the call voice data is divided into multiple segments of call voice data according to the multiple semantic scenes; the multiple segments of call voice data are stored as multiple standardized statement time sequence mapping subsequences respectively; and the response parameters of the call voice data are fed back according to multiple call processing duration parameters corresponding to each standardized statement time sequence mapping subsequence.

[0011] In a possible implementation, after obtaining multiple semantic scenarios, multiple effective query sequence lengths are configured according to the multiple semantic scenarios; based on the multiple effective query sequence lengths, when calculating each normalized statement temporal mapping subsequence at each self-attention head of each encoder layer, the sequence is pruned according to the corresponding effective query sequence length.

[0012] In a possible implementation, the gating weights of the multi-head self-attention mechanism are generated based on the multiple semantic scenarios; the linear mapping matrix calculated for each semantic scenario is corrected according to the gating weights to generate an attention distribution corresponding to the multiple semantic scenarios; and the encoding learning results are updated according to the attention distribution to obtain updated multiple call processing duration parameters.

[0013] In a second aspect, this application provides a speech recognition-based intelligent analysis and processing system for call data. The system comprises: a call voice data collection module for collecting call voice data, performing sentence recognition and timestamp annotation on the call voice data to obtain a sentence time-series mapping sequence; a vector extraction module for calculating interruption durations on the sentence time-series mapping sequence, storing it as a standardized sentence time-series mapping sequence, and extracting semantic feature standardized vectors and interruption feature standardized vectors from the standardized sentence time-series mapping sequence; an encoding learning module for pre-training a Transformer model, where a multi-layer encoder of the Transformer model performs multi-head self-attention encoding learning on the semantic feature standardized vectors and interruption feature standardized vectors, and inputs the encoding learning results into a decoder connected to the encoder, the decoder being used to decode and obtain multiple call processing duration parameters; and a feedback processing module for performing feedback processing on the response parameters of the call voice data based on the multiple call processing duration parameters.

[0014] In summary, one or more technical solutions provided in this application construct a standardized temporal mapping sequence containing semantic features and interruption features, capture long-distance dependencies and multi-granular rhythmic changes in interactions, combine a gating attention mechanism, accurately decode multiple call processing duration parameters through a Transformer encoder-decoder structure, dynamically adjust response parameters, and achieve the technical effect of intelligent closed-loop feedback control of the call service process. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0016] Figure 1 A flowchart illustrating the intelligent analysis and processing method for call data based on speech recognition is provided for this application.

[0017] Figure 2 This application provides a schematic diagram of the structure of a call data intelligent analysis and processing system based on speech recognition.

[0018] Explanation of reference numerals in the attached diagram: Call voice data collection module M100, vector extraction module M200, code learning module M300, feedback processing module M400. Detailed Implementation

[0019] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. It should be understood that this application is not limited to the exemplary embodiments described herein. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application. It should also be noted that, for ease of description, only the parts related to this application are shown in the accompanying drawings, not all of them.

[0020] Example 1: The present application will be described in detail below with reference to the accompanying drawings, as follows... Figure 1 As shown, this application provides a method for intelligent analysis and processing of call data based on speech recognition, wherein the method includes: S1: Collect call voice data, perform sentence recognition and timestamp annotation on the call voice data to obtain a sentence time sequence mapping sequence; S2: Calculate the interruption duration of the sentence time sequence mapping sequence, store it as a standardized sentence time sequence mapping sequence, and extract the semantic feature standardization vector and interruption feature standardization vector of the standardized sentence time sequence mapping sequence.

[0021] Specifically, collecting call voice data refers to the real-time acquisition of the two-channel voice stream between the user and the agent through the call center system, typically using a PCM encoding format with a sampling rate of 8kHz / 16kHz and a quantization precision for digital storage; sentence recognition is the process of segmenting continuous speech into independent sentence units based on automatic speech recognition technology, involving voice activity detection to determine the start and end points of the speech, speaker separation to distinguish between the user and the agent's turn, and speech recognition to convert speech into text; timestamp annotation is to mark the absolute time and relative timing relationship for each identified sentence unit. For example, the absolute time is marked as 3.5 seconds to 8.2 seconds after the start of the call, and the relative timing relationship includes the turn number and the speaker role; the sentence timing mapping sequence is an ordered data structure composed of a sentence text, a start timestamp, an end timestamp, and a speaker role quadruple.

[0022] Interruption duration calculation quantifies the time interval between adjacent turns, including response delay, silence duration, and overlap duration. Response delay refers to the interval between one party ending their speech and another beginning theirs; silence duration refers to the period of silence during which neither party speaks; and overlap duration refers to the period of simultaneous speaking by both parties. The standardized sentence temporal mapping sequence is a preprocessing result that converts the original timestamps into relative time coordinates and normalizes the interruption duration to a unified dimension. The semantic feature standardization vector extracts multi-dimensional semantic embedding vectors from the sentence text using a pre-trained language model, and normalizes them to a magnitude of 1 using the L2 norm. The interruption feature standardization vector is a statistical feature vector calculated from the temporal mapping sequence, containing indicators such as response delay, silence ratio, turn-switching frequency, and interruption mode category, and is standardized using Z-score to eliminate dimensional differences.

[0023] Execution steps: Transcribe the original call speech and simultaneously output the start and end timestamps of each statement to form a preliminary statement time sequence mapping. For example, in a 30-second customer service dialogue, six alternating statements may be identified, each with a time interval such as [0.8s–2.1s] or [2.5s–4.0s]. Calculate the interval between adjacent statements and distinguish speaker roles, including users and agents, to extract multiple types of interruption features. These original interruption values ​​are normalized by Z-score or Min-Max and, together with the semantic vector, constitute a dual-channel input. Preferably, unstructured speech data is transformed into structured, standardized temporal representations that integrate semantic and rhythmic information, enhancing the ability to depict complex dialogue dynamics and providing high-quality, comparable input features for the Transformer model.

[0024] S3: In the pre-trained Transformer model, the multi-layer encoder of the Transformer model performs multi-head self-attention mechanism encoding learning on the semantic feature normalization vector and the interruption feature normalization vector, and inputs the encoding learning result into the decoder connected to the encoder. The decoder is used to decode and obtain multiple call processing duration parameters; S4: Feedback processing is performed on the response parameters of the call voice data according to the multiple call processing duration parameters.

[0025] Specifically, a pre-trained Transformer model refers to a deep neural network architecture based on a self-attention mechanism. Its core components include multi-layered stacked encoders and decoders. It is pre-trained using large-scale unsupervised corpora for masked language models or causal language models, and learns general semantic representations before transferring them to downstream tasks. The multi-layered encoder is usually composed of 6-12 layers of identical encoding blocks, each layer containing a multi-head self-attention mechanism and a feedforward neural network, responsible for extracting high-order abstract representations from the input features layer by layer. The multi-head self-attention mechanism encodes and learns by mapping semantic feature vectors and interruption feature vectors to query, key, and value vector feature matrices. It calculates the dependency weights between different time steps by scaling dot product attention, enabling the model to simultaneously focus on the association pattern between the current sentence semantics and the historical interaction rhythm. The multi-head mechanism allows different attention heads to focus on specific duration patterns, such as user response deadline duration, system processing waiting time, and feedback rhythm changes.

[0026] The encoding learning result is a context-aware feature representation processed by a multi-layer encoder, which integrates long-distance temporal dependencies and multi-granularity behavioral patterns. The decoder is a neural network structure symmetrical to the encoder, which maps the encoding result to the target output space through masked self-attention and cross-attention mechanisms. Multiple call processing duration parameters include: user voice cutoff response duration, call data processing waiting duration, and call feedback rhythm change duration. Furthermore, the user voice cutoff response duration refers to the remaining time required for the user to finish expressing their current intent, the call data processing waiting duration refers to the time required for the system / agent to prepare an effective response, and the call feedback rhythm change duration refers to the prediction of the trend of acceleration or deceleration of the interaction rhythm. The response parameter feedback processing dynamically adjusts the call system's response strategy based on the predicted duration parameters, such as adaptive silence detection threshold, preloading response content, and dynamically adjusting the speech synthesis rate.

[0027] Execution steps: The standardized semantic feature vector and interruption feature vector are concatenated or fed into the pre-trained Transformer encoder via a cross-modal fusion layer. For example, for a dialogue with 10 turns, each turn corresponds to a 768-dimensional semantic vector and a 32-dimensional interruption vector, which are fused to form a unified input sequence. The multi-layer encoder models in parallel using a multi-head self-attention mechanism: some attention heads focus on semantic coherence, while others capture long-distance interruption patterns. The contextual representation output by the encoder is fed into the decoder, which simultaneously predicts multiple call processing duration parameters using a multi-task learning framework. For example, in a bank customer service scenario, the model can simultaneously output: the maximum waiting time before the user abandons the call, the agent's effective response window, and the system's silence tolerance threshold. These parameters are then used to dynamically adjust the IVR process. If the predicted user tolerance time is lower than the current queue waiting estimate, the call is immediately escalated to a human agent or a callback SMS is sent. Through these steps, differentiated user interaction patterns are identified, improving the accuracy of user intent recognition.

[0028] Furthermore, the semantic feature normalization vector and the interruption feature normalization vector are encoded and learned by a multi-head self-attention mechanism using a multi-layer encoder of the Transformer model. The method of this application includes: Each layer of the multi-layer encoder includes a multi-head self-attention mechanism, a feedforward network layer, and a residual network layer. The multi-head self-attention mechanism includes multiple attention heads for learning user voice cutoff response duration features, call data processing waiting duration features, and call feedback rhythm change duration features. The semantic feature normalization vector and the interruption feature normalization vector are concatenated to obtain a concatenated feature normalization vector. The multi-head self-attention mechanism is then used to encode and learn the concatenated feature normalization vector.

[0029] Specifically, the feedforward network layer is a fully connected neural network in the Transformer encoder, following the self-attention mechanism. It consists of two linear transformation layers and an intermediate nonlinear activation function. Its role is to perform position-by-position independent nonlinear transformations on the features extracted by the attention mechanism, enhancing the model's expressive power and feature dimension transformation capabilities. The residual network layer refers to a combination of residual connections and layer normalization, alleviating the gradient vanishing problem in deep networks and enabling the model to be stably trained to the required depth. The concatenated feature normalization vector is a vector concatenation operation between the semantic feature normalization vector and the interrupt feature normalization vector according to the feature dimension, forming a multi-dimensional joint feature representation, so that the input at each time step simultaneously contains what... Dual modal information of "what" and "how"; multiple attention heads refer to the structure in multi-head self-attention mechanisms that projects input features onto multiple low-dimensional subspaces and computes attention in parallel. Typically, 8-12 attention heads are set up, with each head allocated a 64-128 dimensional projection space. Specific heads learn user voice cutoff response duration features, call data processing wait duration features, and call feedback rhythm change duration features through task-oriented pre-training or structured sparse constraints. Among these, user voice cutoff response duration features focus on the integrity signal of speaking intent, call data processing wait duration features focus on the correlation pattern between semantic complexity and system load, and call feedback rhythm change duration features focus on the dynamic pattern of interaction rhythm transitions.

[0030] Execution steps: The semantic feature vector and interruption feature vector corresponding to each turn are concatenated into a concatenated feature normalization vector to form a complete input sequence. The concatenated feature normalization vector layer is processed by a multi-layer encoder. In each layer, the multi-head self-attention mechanism is explicitly configured as a task-oriented structure. Furthermore, in a 12-head setup, four heads can be assigned to specifically focus on the user's voice cutoff response time, focusing on the timestamp difference feature between the end of the user's sentence and the start of the next sentence with high weights; four heads model and process the waiting time to identify abnormal silence on the system side, such as delays caused by ASR post-processing or CRM queries; the remaining four heads capture feedback. Rhythmic variations are achieved by tracking the trend of response interval changes across turn attention, such as increasing from 1.2s to 3.5s. This functional differentiation enables the model to extract multi-granularity and multi-source temporal dependencies in parallel, avoiding information mixing in complex interactions by a single attention head. Residual connections and LayerNorm ensure the stability of deep training, while the feedforward network enhances nonlinear expressive power. Preferably, compared with the standard uniform multi-head setting, the structured attention mechanism accurately decouples the duration behavior patterns under different interaction intentions, achieving deep fusion of semantic and interruption features, and providing a strong representational basis for subsequent high-precision decoding of multiple call processing duration parameters.

[0031] Furthermore, the method of this application utilizes the multi-head self-attention mechanism to perform multi-head self-attention mechanism encoding learning on the concatenated feature normalization vector. Each attention head of the multi-head self-attention mechanism is used to calculate a linear mapping matrix based on the concatenated feature normalization vector. The linear mapping matrix includes a key vector feature matrix, a value vector feature matrix, and a query vector feature matrix. Attention weights are calculated on the linear mapping matrix to obtain an attention weight matrix. The weighted output features of the multiple attention weight matrices are concatenated by multiple heads to obtain a multi-head self-attention concatenated output sequence. The multi-head self-attention concatenated output sequence is processed according to the feedforward network layer and the residual network layer to obtain a multi-head self-attention encoded output sequence. The multi-head self-attention encoded output sequence is output as the encoding learning result.

[0032] Specifically, the linear mapping matrix refers to the linear transformation operation that projects input features from the original dimension to a lower-dimensional subspace through a learnable weight matrix. In multi-head self-attention mechanisms, this is manifested as three types of parameter matrices: query projection matrix, key projection matrix, and value projection matrix. Furthermore, the key vector feature matrix stores the location and attribute information of the queried content and is used to calculate similarity with the query vector to determine the attention distribution. Each row corresponds to the key vector representation of a time step. The value vector feature matrix stores the actual information content to be aggregated and forms the output after being weighted by attention weights. It is the main carrier of feature transmission. The query vector feature matrix represents the retrieval request of the current time step. By calculating the dot product matching degree with all key vectors, it determines which historical positions the model should focus on.

[0033] Attention weight matrix A i It is a square matrix obtained by scaling and Softmax normalization after multiplying the query projection matrix and the transpose of the key projection matrix. The attention weight matrix satisfies the normalization constraint that the sum of probabilities in each row is 1. Multi-head concatenation refers to concatenating the output matrices of multiple attention heads along the feature dimension to form a joint representation that integrates the attention patterns of multiple subspaces. This concatenation result constitutes the multi-head self-attention concatenation output sequence. The multi-head self-attention concatenation output sequence is then passed through the nonlinear transformation and residual connection of the feedforward network layer to finally obtain the multi-head self-attention encoding output sequence, which is output as the encoding learning result and has strong context awareness and enhanced feature representation ability.

[0034] Execution steps: For the input concatenated feature normalization vector sequence X∈R n×d Each attention head h i Using an independent learnable weight matrix W Q i W K W V i ∈ Calculate Q respectively i K i V i Qi =XW Q i K i =W K i V i =W V i The attention weight matrix A is calculated using the scaled dot product attention formula. i =Softmax( Attention i =A i V i =Softmax( V i .

[0035] Where X represents the concatenated feature normalization vector sequence input to the Transformer encoder, i.e., the structured data after fusing semantic features and interruption features; R n×d Let X be a real matrix with n rows and d columns; n represents the number of turns in the call dialogue, i.e. the total number of statements spoken by the user and the agent; d represents the dimension of the feature vector corresponding to each turn.

[0036] Among them, h i W represents the i-th attention head, where i = 1, 2, ..., H, and H is the total number of attention heads; Q i W is the query projection matrix for the i-th attention head, used to map input features to the query subspace of that head; K i W is the key projection matrix for the i-th attention head, used to generate the key vector for similarity calculation; V i The projection matrix for the value of the i-th attention head is used to generate the information content to be aggregated; This indicates that these weight matrices are all d rows. A real matrix of columns; The subspace dimension within each attention head is typically set to 1. =d / H, for example, when d=768 and H=12, =64, this design ensures that the output of all heads can be stitched together to restore the original dimensions or adapt to the hidden layer size of the model.

[0037] Among them, Q i =XW Q i , represents the query vector feature matrix of the i-th head; K i =W K i , represents the key vector feature matrix of the i-th head; Vi =W V i , represents the feature matrix of the value vector of the i-th head.

[0038] in, This represents the dot product matrix of the query and the key, with dimension R. n×n ; Softmax represents the scaling factor, used to alleviate gradient saturation problems caused by high-dimensional dot products; Normalize each row to make the result a probability distribution, i.e., A i ∈R n×n ,satisfy =1, A i It is an n×n matrix corresponding to the i-th query position. Specifically, A represents the probability of attention from the i-th turn to the j-th turn; i V i Attention weights are applied to the feature matrix of the value vector to achieve weighted aggregation, resulting in the output of the i-th head, with the dimension still being [dimensionality missing]. .

[0039] For example, in a 12-head setup, each head outputs a dimension of 64. The outputs of the 12 heads are concatenated to restore a 768-dimensional array. This concatenated result is then processed by a feedforward network, and residual connections and LayerNorms are introduced into both the self-attention and FFN sub-layers to form a complete encoder output. This achieves deep contextual reweighting of semantic-interruption fusion features. Specifically, when a user speaks three times consecutively, the agent's response delay increases. The attention head responsible for feedback rhythm changes will assign higher weights to later turns in Query-Key matching, thereby highlighting the rhythm deterioration signal in Value aggregation. Through the above steps, the output multi-head self-attention encoded output sequence is used as a high-order fusion representation and fed into the decoder to accurately decode multiple call processing duration parameters, supporting subsequent intelligent feedback control.

[0040] Furthermore, after obtaining the multi-head self-attention encoding learning results, the method of this application includes: The multi-head self-attention encoded output sequence is sent to the decoder, which includes multiple prediction heads corresponding to the multi-head self-attention mechanism for predicting the user's voice cutoff response duration, the learning call data processing waiting duration, and the call feedback rhythm change duration. The encoders with mean square errors of the multiple prediction heads less than a preset threshold are set to the convergence state, thereby obtaining multiple call processing duration parameters decoded by the decoder when it is in the convergence state.

[0041] Specifically, the decoder refers to the neural network component in the Transformer architecture responsible for mapping the context-aware feature sequence output by the encoder to the target prediction result. Unlike the encoder's bidirectional attention, the decoder uses masked self-attention to ensure that position i can only rely on the history on the left, and introduces the encoder output as a key / value source through a cross-attention layer to realize the interaction of encoding and decoding information. Multiple prediction heads are the task-specific output layers at the top of the decoder. Each prediction head is an independent branch composed of 2-3 layers of fully connected networks, corresponding to the prediction of user voice cutoff response duration, call data processing waiting time, and call feedback rhythm change duration, forming a multi-task learning architecture. The user voice cutoff response duration refers to the predicted time interval from the current moment to the user completing the current intention expression and naturally ending the speech. It is a key parameter for controlling the timing of agent takeover and avoiding interruption.

[0042] Call data processing wait time refers to the predicted time interval from when the user ends their speech to when the system / agent is ready and issues a valid response. It is used to dynamically adjust the waiting prompt strategy and resource preloading. Call feedback rhythm change time refers to the predicted time window and direction of change when the interaction rhythm changes significantly relative to the historical benchmark. It supports dialogue fluency management. Mean squared error is a loss function that measures the difference between the predicted value and the true labeled value. The preset threshold is an empirical hyperparameter for determining model convergence. It is usually set as the inflection point value where the validation set MSE no longer decreases or the maximum error tolerance acceptable to the task. Convergence state means that the model parameter optimization has reached a stable equilibrium, and the validation loss has decreased by less than the threshold for multiple consecutive rounds without obvious signs of overfitting.

[0043] Execution steps: The multi-head self-attention encoded output sequence from the encoder is fed into the decoder. The decoder first further refines the task-related contextual information through its internal self-attention and encoder-decoder cross-attention layers. Then, the final hidden state is input into three independent prediction heads: each prediction head typically consists of 1-2 layers of fully connected networks, outputting a continuous duration value. For example, prediction head 1 outputs the user's voice cutoff response duration, prediction head 2 outputs the call data processing waiting time, and prediction head 3 outputs the feedback rhythm change slope. During the training phase, the mean squared error (MSE) of each prediction head's output is calculated with its corresponding ground truth label, and the total loss is the weighted sum of the MSEs of each head. When all MSEs are below a preset threshold for several consecutive rounds, the model is considered to have entered a convergence state. At this point, the decoder outputs multiple call processing duration parameters. Preferably, the task-aligned multi-predictor head design and strict convergence criteria ensure that the decoded duration parameters are both accurate and business-operable, thus supporting subsequent precise response parameter feedback control.

[0044] Furthermore, after collecting call voice data, the method of this application also includes: The call voice data is subjected to semantic scene recognition to obtain multiple semantic scenes; the call voice data is divided into multiple segments of call voice data according to the multiple semantic scenes; the multiple segments of call voice data are stored as multiple standardized statement time sequence mapping subsequences respectively; and the response parameters of the call voice data are fed back according to multiple call processing duration parameters corresponding to each standardized statement time sequence mapping subsequence.

[0045] Specifically, semantic scene recognition refers to the classification process based on understanding the topic and business intent of a dialogue based on the speech content. By using a pre-trained language model to perform intent recognition and topic segmentation on the transcribed text of a call, continuous calls are divided into semantic units with clear business boundaries. Unlike simple keyword matching, semantic scene recognition integrates contextual information and domain knowledge, and supports nested scenarios and hierarchical classification. For example, semantic units with clear business boundaries include scenario categories such as identity verification, business consultation, complaint handling, and transaction processing. Multiple semantic scenes are heterogeneous business unit sequences that appear in chronological order within a single call. For example, a complete call sequentially goes through four semantic scenes: "greeting opening → account inquiry → transfer processing → ending farewell," each with different interaction norms and duration characteristics.

[0046] Multi-segment call voice data are sub-segments obtained by time-axis segmentation of the original continuous voice stream according to semantic scene boundaries. Each segment corresponds to a single semantic scene, maintaining internal semantic coherence and external scene independence. The standardized statement temporal mapping sub-sequence is an independent temporal feature sequence generated after processing each voice data segment in step one. It includes statement timestamps, semantic feature vectors, and interruption feature vectors within the scene. The sequence length varies depending on the scene complexity. Separate storage means persisting each sub-sequence as an independent data structure, supporting parallel processing and scene-level index retrieval, avoiding computational redundancy and information confusion caused by uniform processing of long sequences.

[0047] Execution steps: The transcribed text of the entire call is semantically segmented. For example, a 5-minute bank customer service call may be identified as three consecutive scenarios: identity verification (0s–45s), credit card limit inquiry (46s–180s), and temporary limit increase application (181s–300s). The system detects scenario switching points based on a pre-trained scenario classifier. Subsequently, the original voice data is segmented into three segments according to these boundaries, and a standardized statement time sequence mapping subsequence is independently constructed for each segment, including recalculating the statement timestamps within the segment, the interruption duration within the segment, and so on. The corresponding semantic and interruption feature vectors are extracted. Then, each sub-sequence is input into the converged Transformer encoder-decoder, which outputs its own call processing time parameters. For example, the predicted agent response tolerance threshold for the identity verification segment is 2.5 seconds, and the predicted agent response tolerance threshold for the credit card limit inquiry segment is 1.4 seconds. Finally, the system implements differentiated feedback based on these segment parameters. For example, in highly sensitive scenarios such as complaints, if the response delay exceeds the predicted threshold in real time, priority queue scheduling or the insertion of reassuring dialogue is immediately triggered. Compared to the limitations of unified modeling for all-connection dialogues, achieving scenario-adaptive, refined duration modeling and dynamic feedback significantly enhances the context awareness and service agility of the intelligent call system.

[0048] Furthermore, the method of this application includes: After obtaining multiple semantic scenarios, multiple effective query sequence lengths are configured according to the multiple semantic scenarios; based on the multiple effective query sequence lengths, when calculating each normalized statement temporal mapping subsequence at each self-attention head of each encoder layer, the sequence is pruned according to the corresponding effective query sequence length.

[0049] Specifically, the effective query sequence length refers to the size of the attention mechanism calculation window, which is dynamically set according to the semantic scene characteristics. This window is used to limit the range of historical context accessible to the query vector in the self-attention mechanism, replacing the fixed global length calculation mode and achieving adaptive adaptation to different scene complexities and information needs. The configuration process converts semantic labels into integer length values ​​based on the scene category mapping table. For example, a short window is configured for simple scenarios such as greetings and openings, while a long window is configured for complex scenarios such as complaints and cross-system processing. This mapping is determined through domain expert rules or data-driven search. The pruning operation is performed before attention calculation, truncating or sliding window processing the historical sequence that exceeds the effective length, reducing the dimension of the key projection matrix / value projection matrix to the effective length, reducing the computational complexity of self-attention, and significantly reducing the computational overhead and memory usage of long scenarios.

[0050] Execution steps: Assign a customized effective query sequence length based on the semantic scenario type. For example, in a customer service system, password reset scenarios usually have short conversations and fixed processes, so the effective length can be configured to 15; while service dispute scenarios have dense interactions and varied rhythms, so it can be configured to 40. This configuration can be obtained through historical data analysis. Furthermore, the distribution of the number of call rounds in each scenario is statistically analyzed and determined in combination with business expert rules.

[0051] In each layer of the Transformer encoder, when processing a subsequence, the system dynamically reads the effective length L corresponding to the semantic scene to which the subsequence belongs and performs pruning on the input sequence: if the original subsequence contains M turns, the first L turns are retained, or key segments are retained based on importance scoring, and the rest are discarded, where M>L. For example, a bill dispute subsequence contains 48 turns, but its effective query length is set to 40, so the last 8 low-information turns are pruned. Low-information turns include repeated confirmation statements. This scene-aware sequence pruning mechanism significantly improves computational efficiency and modeling focus. Furthermore, because it removes noise caused by redundant silences or repeated confirmations, the pruning operation is performed independently in each layer and each attention head, ensuring that different functional heads work within the optimal context window. In the above steps, by introducing semantic scene-driven dynamic sequence length control, the collaborative optimization of computational resources and modeling accuracy is achieved, providing scalable technical support for high-concurrency call analysis systems.

[0052] Furthermore, after acquiring multiple semantic scenarios, the method of this application also includes: Gating weights for the multi-head self-attention mechanism are generated based on the multiple semantic scenarios; the linear mapping matrix calculated for each semantic scenario is corrected according to the gating weights to generate an attention distribution corresponding to the multiple semantic scenarios; the encoding learning results are updated according to the attention distribution to obtain updated multiple call processing duration parameters.

[0053] Specifically, gating weights refer to scalar coefficient vectors dynamically generated based on the semantic scene, used to regulate the activation intensity of different attention heads in the multi-head self-attention mechanism, achieving scene-adaptive feature selection. Furthermore, for each identified semantic scene, through learnable scene embedding and gating networks, a weight vector with a dimension equal to the number of attention heads is generated for each scene category. The closer the value of the weight vector is to 1, the more important the head is in the current scene; the closer it is to 0, the more suppressed the contribution of the head is. The linear mapping matrix is ​​corrected. Specifically, the gating weights are applied to the query vector feature matrix, key vector feature matrix, or value vector feature matrix generated by each attention head in a head-by-head multiplication manner, thereby adjusting their representation intensity. Attention distribution refers to the attention probability matrix between time steps after gating correction, and its numerical distribution reflects the scene-specific information filtering strategy. Update refers to re-inputting the gating-corrected attention distribution into the feedforward network and residual layer to generate a corrected version of the encoding learning result, which is finally transmitted to the decoder to output the duration parameter adapted to the scene.

[0054] Execution steps: A learnable scene embedding vector is assigned to each semantic scene. This vector is mapped to a gating weight vector corresponding to the number of attention heads through a small gating network. Subsequently, after calculating the linear mapping matrix of each head, the system multiplies the gating weights element by element by the query vector feature matrix / key vector feature matrix / value vector feature matrix of the corresponding head, realizing dynamic enhancement or attenuation of feature channels. The resulting attention distribution is more focused on the key behavioral patterns of the current scene. The context representation output by the encoder is updated by this gating attention mechanism and then sent to the decoder to re-predict the call processing time parameter. Preferably, this mechanism enables the model to have scene-aware attention bias capability, avoiding the averaging bias of general attention in complex multi-scene environments, thereby outputting more accurate and business-interpretive update parameters, providing a reliable basis for differentiated response strategies.

[0055] In summary, the beneficial effects of the embodiments of this application are: By collecting call voice data, performing sentence recognition and timestamp annotation on the call voice data to obtain a sentence time-series mapping sequence; calculating the interruption duration of the sentence time-series mapping sequence and storing it as a standardized sentence time-series mapping sequence; extracting the semantic feature standardized vector and interruption feature standardized vector from the standardized sentence time-series mapping sequence; in the pre-trained Transformer model, the multi-layer encoder of the Transformer model performs multi-head self-attention mechanism encoding learning on the semantic feature standardized vector and interruption feature standardized vector, and inputting the encoding learning result into the decoder connected to the encoder, the decoder is used to decode and obtain multiple call processing duration parameters; and performing feedback processing on the response parameters of the call voice data based on multiple call processing duration parameters, this application provides a method and system for intelligent analysis and processing of call data based on speech recognition, constructs a standardized time-series mapping sequence containing semantic features and interruption features, captures long-distance dependencies and multi-granular rhythmic changes in the interaction, and combines a gating attention mechanism to accurately decode multiple call processing duration parameters through a Transformer encoder-decoder structure, dynamically adjusts the response parameters, and achieves the technical effect of intelligent closed-loop feedback control of the call service process.

[0056] Example 2 is based on the same inventive concept as the speech recognition-based intelligent analysis and processing method for call data in the foregoing examples, such as... Figure 2 As shown in the figure, this application provides a call data intelligent analysis and processing system based on speech recognition, wherein the system includes: Call voice data collection module M100: collects call voice data, performs sentence recognition and timestamp annotation on the call voice data, and obtains a sentence time sequence mapping sequence.

[0057] Vector extraction module M200: Calculates the interruption duration of the statement timing mapping sequence, stores it as a standardized statement timing mapping sequence, and extracts the semantic feature standardized vector and interruption feature standardized vector of the standardized statement timing mapping sequence.

[0058] Encoding learning module M300: In the pre-trained Transformer model, the multi-layer encoder of the Transformer model performs multi-head self-attention mechanism encoding learning on the semantic feature normalization vector and interruption feature normalization vector. The encoding learning result is input into the decoder connected to the encoder. The decoder is used to decode and obtain multiple call processing duration parameters.

[0059] Feedback processing module M400: Performs feedback processing on the response parameters of the call voice data based on the multiple call processing duration parameters.

[0060] Furthermore, the encoding learning module M300 is used to perform the following methods: Each layer of the multi-layer encoder includes a multi-head self-attention mechanism, a feedforward network layer, and a residual network layer. The multi-head self-attention mechanism includes multiple attention heads for learning user voice cutoff response duration features, call data processing waiting duration features, and call feedback rhythm change duration features. The semantic feature normalization vector and the interruption feature normalization vector are concatenated to obtain a concatenated feature normalization vector. The multi-head self-attention mechanism is then used to encode and learn the concatenated feature normalization vector.

[0061] Furthermore, the encoding learning module M300 is also used to perform the following methods: Each attention head of the multi-head self-attention mechanism is used to calculate a linear mapping matrix based on the concatenated feature normalization vector. The linear mapping matrix includes a key vector feature matrix, a value vector feature matrix, and a query vector feature matrix. Attention weights are calculated on the linear mapping matrix to obtain an attention weight matrix. The weighted output features of the multiple attention weight matrices are concatenated by multiple heads to obtain a multi-head self-attention concatenated output sequence. The multi-head self-attention concatenated output sequence is processed according to the feedforward network layer and the residual network layer to obtain a multi-head self-attention encoded output sequence. The multi-head self-attention encoded output sequence is output as the encoding learning result.

[0062] Furthermore, the encoding learning module M300 is also used to perform the following methods: The multi-head self-attention encoded output sequence is sent to the decoder, which includes multiple prediction heads corresponding to the multi-head self-attention mechanism for predicting the user's voice cutoff response duration, the learning call data processing waiting duration, and the call feedback rhythm change duration. The encoders with mean square errors of the multiple prediction heads less than a preset threshold are set to the convergence state, thereby obtaining multiple call processing duration parameters decoded by the decoder when it is in the convergence state.

[0063] Furthermore, the call voice data collection module M100 is also used to perform the following methods: The call voice data is subjected to semantic scene recognition to obtain multiple semantic scenes; the call voice data is divided into multiple segments of call voice data according to the multiple semantic scenes; the multiple segments of call voice data are stored as multiple standardized statement time sequence mapping subsequences respectively; and the response parameters of the call voice data are fed back according to multiple call processing duration parameters corresponding to each standardized statement time sequence mapping subsequence.

[0064] Furthermore, the call voice data collection module M100 is also used to perform the following methods: After obtaining multiple semantic scenarios, multiple effective query sequence lengths are configured according to the multiple semantic scenarios; based on the multiple effective query sequence lengths, when calculating each normalized statement temporal mapping subsequence at each self-attention head of each encoder layer, the sequence is pruned according to the corresponding effective query sequence length.

[0065] Furthermore, the call voice data collection module M100 is also used to perform the following methods: Gating weights for the multi-head self-attention mechanism are generated based on the multiple semantic scenarios; the linear mapping matrix calculated for each semantic scenario is corrected according to the gating weights to generate an attention distribution corresponding to the multiple semantic scenarios; the encoding learning results are updated according to the attention distribution to obtain updated multiple call processing duration parameters.

[0066] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Figure 1 The intelligent analysis and processing method and specific examples of call data based on speech recognition in Embodiment 1 are also applicable to the intelligent analysis and processing system of call data based on speech recognition in this embodiment. Through the foregoing detailed description of the intelligent analysis and processing method of call data based on speech recognition, those skilled in the art can clearly understand the intelligent analysis and processing system of call data based on speech recognition in this embodiment. Therefore, for the sake of brevity, it will not be described in detail here.

[0067] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0068] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of this application and its equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for intelligent analysis and processing of call data based on speech recognition, characterized in that, The method includes: Collect call voice data, perform sentence recognition and timestamp annotation on the call voice data, and obtain a sentence time sequence mapping sequence; The interruption duration is calculated for the statement timing mapping sequence and stored as a standardized statement timing mapping sequence. The semantic feature standardized vector and interruption feature standardized vector of the standardized statement timing mapping sequence are extracted. The interruption duration calculation is a quantitative analysis of the time interval between adjacent turns, including response delay, silence duration and overlap duration. In the pre-trained Transformer model, the multi-layer encoder of the Transformer model performs multi-head self-attention mechanism encoding learning on the semantic feature normalization vector and the interruption feature normalization vector. The encoding learning result is input into the decoder connected to the encoder. The decoder is used to decode and obtain multiple call processing duration parameters, including user voice cutoff response duration, call data processing waiting duration, and call feedback rhythm change duration. Feedback processing is performed on the response parameters of the call voice data based on the multiple call processing duration parameters. The response parameter feedback processing is to dynamically adjust the response strategy of the call system based on the predicted duration parameters.

2. The method as described in claim 1, characterized in that, The semantic feature normalization vector and the interruption feature normalization vector are encoded and learned by a multi-head self-attention mechanism using a multi-layer encoder of the Transformer model. The method includes: Each layer of the multi-layer encoder includes a multi-head self-attention mechanism, a feedforward network layer, and a residual network layer. The multi-head self-attention mechanism includes multiple attention heads for learning user voice cutoff response duration features, call data processing waiting duration features, and call feedback rhythm variation duration features. The semantic feature normalization vector and the interruption feature normalization vector are concatenated to obtain the concatenated feature normalization vector. The concatenated feature normalization vector is then encoded and learned using the multi-head self-attention mechanism.

3. The method as described in claim 2, characterized in that, The method for learning multi-head self-attention mechanism encoding on the concatenated feature normalization vector using the multi-head self-attention mechanism includes: Each attention head of the multi-head self-attention mechanism is used to calculate a linear mapping matrix based on the concatenated feature normalization vector, the linear mapping matrix including a key vector feature matrix, a value vector feature matrix, and a query vector feature matrix; The attention weight matrix is ​​obtained by calculating the attention weights on the linear mapping matrix. The weighted output features of the multiple attention weight matrices are concatenated by multiple heads to obtain a multi-head self-attention concatenated output sequence; The multi-head self-attention concatenation output sequence is processed according to the feedforward network layer and the residual network layer to obtain the multi-head self-attention encoded output sequence, and the multi-head self-attention encoded output sequence is output as the encoding learning result.

4. The method as described in claim 3, characterized in that, After obtaining the results of multi-head self-attention encoding learning, the methods include: The multi-head self-attention encoded output sequence is sent to the decoder, which includes multiple prediction heads corresponding to the multi-head self-attention mechanism for predicting the user's voice cutoff response duration, call data processing waiting duration, and call feedback rhythm change duration. When the mean square error of the multiple prediction heads is less than a preset threshold, the encoder is set to the convergence state, and multiple call processing time parameters of the decoder output when the decoder is in the convergence state are obtained.

5. The method as described in claim 1, characterized in that, After collecting call voice data, the method also includes: Perform semantic scene recognition on the call voice data to obtain multiple semantic scenes; The call voice data is divided into multiple segments of call voice data according to the multiple semantic scenarios; The multiple segments of call voice data are stored as multiple standardized statement timing mapping sub-sequences, and the response parameters of the call voice data are fed back according to multiple call processing duration parameters corresponding to each standardized statement timing mapping sub-sequence.

6. The method as described in claim 5, characterized in that, After obtaining multiple semantic scenarios, configure multiple valid query sequence lengths based on the multiple semantic scenarios; Based on the lengths of the multiple valid query sequences, when calculating each normalized statement temporal mapping subsequence at each self-attention head of each encoder layer, the sequence is pruned according to the corresponding length of the valid query sequence.

7. The method as described in claim 5, characterized in that, After obtaining multiple semantic scenarios, the method also includes: The gating weights of the multi-head self-attention mechanism are generated based on the multiple semantic scenarios; The linear mapping matrix calculated for each semantic scene is corrected according to the gating weights to generate an attention distribution corresponding to the multiple semantic scenes; The encoding learning results are updated according to the attention distribution to obtain multiple updated call processing duration parameters.

8. A call data intelligent analysis and processing system based on speech recognition, characterized in that, The system is used to implement the intelligent analysis and processing method for call data based on speech recognition as described in any one of claims 1-7, wherein the system comprises: Call voice data collection module: collects call voice data, performs sentence recognition and timestamp annotation on the call voice data, and obtains a sentence time sequence mapping sequence; Vector extraction module: Calculates the interruption duration of the statement timing mapping sequence, stores it as a standardized statement timing mapping sequence, and extracts the semantic feature standardized vector and interruption feature standardized vector of the standardized statement timing mapping sequence. The interruption duration calculation is a quantitative analysis of the time interval between adjacent turns, including response delay, silence duration and overlap duration. Encoding Learning Module: In the pre-trained Transformer model, the multi-layer encoder of the Transformer model performs multi-head self-attention mechanism encoding learning on the normalized semantic feature vector and the normalized interruption feature vector. The encoding learning result is input into the decoder connected to the encoder. The decoder is used to decode and obtain multiple call processing duration parameters, including user voice cutoff response duration, call data processing waiting duration, and call feedback rhythm change duration. Feedback processing module: performs feedback processing on the response parameters of the call voice data based on the multiple call processing duration parameters. The response parameter feedback processing dynamically adjusts the call system's response strategy based on the predicted duration parameters.

Images