Double-path voice stream real-time identification method, system and application

Abstract

The invention discloses a double-channel voice stream real-time identification method, system and application, and the method comprises the steps: carrying out the preprocessing of collected VOIP call double-channel audio, and maintaining the time sequence synchronization; extracting Mel-frequency cepstral coefficient features and speech spectrogram features of the preprocessed audio, and inputting the spliced features into a Transform deep neural network model for stream speech recognition to obtain a two-way character sequence; generating a unique identifier based on channel identification and voice energy difference, and establishing a corresponding relation with the character sequence; carrying out punctuation prediction and text standardization by utilizing an LSTM-based model, sorting and aligning character sequences according to timestamp fields, and generating a time sequence dialogue stream; and performing anomaly detection and / or storage management on the time sequence dialogue stream to realize real-time quality inspection and agent assistance. According to the invention, synchronous recognition and role distinguishing of double-channel voice are realized, the recognition delay is low, and the recognition accuracy, the detection precision and the real-time performance are high; and high-efficiency management and safety compliance of data are realized by combining distributed encryption storage.

Description

Technical Field

[0001] This invention relates to the technical field of speech recognition and intelligent quality inspection in call centers, and in particular to a method, system and application for real-time recognition of dual-channel speech streams. Background Technology

[0002] In the fields of speech recognition and intelligent quality inspection in call centers, the development of dual-stream real-time speech recognition technology has always revolved around the three core requirements of "synchronization, real-time performance, and intelligence," evolving from single-stream to dual-stream and then to intelligent real-time processing. Early single-stream speech recognition could only process a single path of speech offline, typically using Hidden Markov Models (HMMs) for speech transcription. Recognition latency reached the second level, and it could not distinguish speaker identity, making it only suitable for post-recording text processing and unable to meet the real-time interaction requirements of VoIP. Subsequently, the nascent dual-stream processing stage began, attempting to simultaneously collect the voices of both customers and agents. However, due to the lack of a precise identity labeling mechanism, the transcription results were easily confused, and the overall system latency exceeded one second. It also lacked anomaly detection capabilities, making it difficult to adapt to real-time quality inspection and decision support scenarios. With the maturity of deep learning and streaming processing architectures, current technological development has entered the stage of intelligent real-time processing. By introducing neural network models such as Transformer, dual-channel speech separation and identity labeling are achieved, and the recognition latency is controlled within 200ms. Furthermore, intelligent anomaly detection algorithms such as sensitive word detection and sentiment analysis are integrated with distributed storage technology to achieve integrated processing of "transcription-analysis-response", which is suitable for various VoIP call center scenarios.

[0003] However, existing technologies still suffer from several technical bottlenecks: First, insufficient dual-channel processing capability; most systems are still based on a single-channel speech recognition architecture, unable to process the voices of both parties in a VoIP call in parallel, leading to identity confusion and affecting the accuracy of quality inspection and agent assistance. Second, poor real-time performance; the end-to-end latency in offline batch processing modes generally exceeds 5 seconds, failing to meet the real-time quality inspection and immediate intervention requirements of call centers. Third, limited anomaly detection capabilities; most solutions rely solely on keyword matching, lacking semantic understanding and sentiment recognition, easily resulting in misjudgments or missed judgments, and failing to effectively detect complex behaviors such as silence timeouts and arguments. Finally, weak data management capabilities; the lack of temporal alignment and indexing retrieval mechanisms for voice and text leads to low efficiency in storing and querying large-scale call data, making it difficult for the system to efficiently adapt to multiple business scenarios. These problems limit the application effectiveness of existing systems in high-concurrency, low-latency, and intelligent analysis scenarios. Summary of the Invention

[0004] To address the aforementioned technical problems in the existing technology, this invention proposes a method, system, and application for real-time recognition of dual-channel speech streams.

[0005] According to a first aspect of the present invention, a real-time dual-channel speech stream recognition method is proposed, comprising: S1: Preprocess the dual-channel audio of the acquired VoIP call and maintain the timing synchronization of the dual-channel audio during the preprocessing stage; S2: Extract speech features from the preprocessed dual-channel audio, including Mel-frequency cepstral coefficient features and speech spectrogram features. Concatenate the speech features and input them into the Transformer deep neural network model for streaming speech recognition to obtain dual-channel text sequences corresponding to the dual-channel audio. Generate unique identifiers based on channel recognition and speech energy difference, and establish the correspondence between the unique identifiers and each text sequence. The unique identifiers include role fields and timestamp fields. S3: Use an LSTM-based model to predict punctuation and normalize text in dual-channel text sequences, and sort and align them according to the timestamp field to generate a temporal dialogue stream corresponding to dual-channel audio. S4: Perform anomaly detection and / or storage management on time-series dialogue streams to enable real-time quality inspection, monitoring, and agent assistance.

[0006] In the above technical solution, real-time recognition and semantic structured output of the voices of both parties in a call are achieved through dual-channel synchronous acquisition, deep neural network streaming speech recognition and dialogue-level text alignment, ensuring the synchronization and integrity of speech transcription, and providing accurate input for subsequent quality inspection and anomaly detection.

[0007] In some specific embodiments, S4 includes a fusion detection mechanism based on rule matching and model analysis to detect anomalies in the temporal dialogue stream. The fusion detection mechanism determines the anomaly level based on the rule score obtained from rule matching and the model score obtained from model analysis, and triggers a corresponding response based on the anomaly level. This fusion detection mechanism achieves multi-dimensional intelligent judgment of abnormal speech behavior by comprehensively analyzing the results of rules and models, effectively improving the accuracy and scene adaptability of anomaly detection.

[0008] In some specific embodiments, rule matching is based on the AC automaton algorithm to perform multi-pattern matching on the text to detect sensitive words and service taboo phrases; model analysis includes semantic violation recognition based on the BERT model, sentiment anomaly analysis based on the LSTM attention mechanism model, and silence detection based on the computation time difference between two adjacent texts. This achieves multi-dimensional, high-precision real-time anomaly recognition and improves system robustness.

[0009] In some specific embodiments, the fusion detection mechanism also includes a timing anomaly determination based on the time difference calculated from the timestamps of two adjacent texts, used to detect silence timeout anomalies in calls; the time difference calculation formula is: In the formula, This is the start timestamp of the next text. This is the end timestamp of the previous text; it responds to the time difference. If the time exceeds the preset threshold, it is judged as a time series anomaly. The time series anomaly score is quantified based on the silence duration and combined with the rule score and model score to re-determine the anomaly level.

[0010] In the aforementioned technical solution, by introducing silence detection based on timestamp differences, "unnatural pauses" in calls can be independently identified, thereby enabling real-time monitoring of the smoothness of customer and agent interactions. Silence anomaly detection is performed in parallel with rule scoring and model scoring, ensuring that the anomaly level is automatically adjusted when service interruptions or response delays are detected during a call. This effectively reduces customer churn caused by call delays or slow agent responses, while simultaneously improving the ability to monitor service quality. By quantifying silence anomalies into scores and combining them with content risk assessment, the system achieves more comprehensive and accurate anomaly detection and real-time response, greatly enhancing the intelligent management level of customer service dialogues and ensuring continuous optimization of the customer experience.

[0011] In some specific embodiments, the LSTM attention mechanism model is built based on an LSTM network and an additive attention mechanism. The LSTM network is used to extract temporal features from the speech-to-text transcription to capture the emotional dependencies of the contextual semantics; the additive attention mechanism is used to calculate the correlation between the hidden state of the LSTM output and the attention weight matrix, and its calculation formula is as follows: In the formula, For a trainable attention weight matrix, A trainable weight matrix is ​​used to train the hidden state vector. Projecting onto the attention space to extract latent emotional intensity features. The bias term is used; the output layer of the LSTM attention mechanism model uses a Softmax classification structure to classify text into three emotion categories: angry, neutral, and positive.

[0012] In the above technical solutions, the LSTM network can model the time-series features of speech-to-text and capture the emotional dependencies between contexts, thereby maintaining the continuity of emotional semantics in long or cross-sentence dialogues; the additive attention mechanism assigns higher weights to key emotional words (such as "anger", "dissatisfaction", "complaint") in the input sequence based on the correlation between the hidden state and the attention weight matrix, thereby achieving significant enhancement and focus of emotional features.

[0013] In some specific embodiments, S4 further includes performing AES-256 symmetric encryption on the time-series dialogue stream before distributed storage. The distributed storage achieves data load balancing based on a consistent hashing algorithm, employs a "one master, two slaves" replica synchronization strategy for redundant data storage, and performs partition management and lifecycle control based on date and business tags. By encrypting and distributing redundant storage of the speech recognition results, secure protection and compliance management of data transmission and storage processes are achieved, ensuring the reliability and traceability of the system in a multi-node environment.

[0014] In some specific embodiments, preprocessing includes adaptive filtering for noise reduction, echo cancellation, and automatic gain control. The adaptive filtering for noise reduction employs a least mean square algorithm, while echo cancellation uses a normalized least mean square algorithm. This preprocessing step, through adaptive filtering and echo suppression, eliminates environmental noise and equipment interference, significantly improving the clarity and recognition accuracy of the speech signal.

[0015] In some specific embodiments, the Mel frequency cepstral coefficient features are extracted using a 25ms window length and a 10ms frame shift to obtain 13th-order Mel frequency cepstral coefficient features. The number of Mel filters is 26, covering the 100Hz–8000Hz frequency band. The formula for calculating the Mel frequency cepstral coefficients is as follows: In the formula, For the first Mehr frequency cepstral coefficients, For the first The output energy of a Mel filter The total number of Mel filters; This represents the order of the cepstral coefficients. The Mel-frequency cepstral coefficient feature extraction method can effectively characterize the spectral envelope and formant characteristics of speech signals, enhancing the system's robustness in recognizing different speakers and accents.

[0016] In some specific embodiments, speech spectrogram features are generated through short-time Fourier transform, with a spectrogram dimension of 128×128. By extracting the time-frequency features of speech, the dynamic changes of different articulatory units can be captured, providing richer input information for deep recognition models.

[0017] In some specific embodiments, S3 further includes building a multi-dimensional inverted index for the generated temporal dialogue stream. The index dimensions of the inverted index include call identifier, timestamp field, role field, and transcribed text keywords. The multi-dimensional inverted index construction process employs an incremental update and periodic merging strategy to balance real-time data writing and full-text retrieval efficiency. By introducing a multi-dimensional inverted index structure into the temporal dialogue stream, rapid location of recognition results and multi-condition retrieval can be achieved. Simultaneously, the index maintenance mechanism combining incremental updates and periodic merging maintains the stability and efficiency of the index structure even with continuously added recognition data, thus balancing the system's real-time data writing capability and full-text retrieval response speed, enabling efficient management and querying of large-scale speech recognition results.

[0018] In some specific embodiments, the attention calculation formula for the Transformer deep neural network model is as follows: In the formula, , , These are the query matrix, key matrix, and value matrix, each with dimensions of [batch size, sequence length, 512]. The dimension of the key matrix, This is a matrix transpose operation. Attention mechanisms can capture long-range dependencies during speech recognition, improving contextual understanding of speech content and recognition accuracy.

[0019] According to a second aspect of the invention, a computer-readable storage medium is provided on which one or more computer programs are stored, which, when executed by a computer processor, implement the method described above.

[0020] According to a third aspect of the present invention, a real-time dual-channel speech stream recognition system is proposed, comprising: The audio preprocessing module is configured to preprocess the dual-channel audio of the acquired VoIP call and maintain the timing synchronization of the dual-channel audio during the preprocessing stage. The speech recognition module is configured to extract speech features from preprocessed dual-channel audio. The speech features include Mel-frequency cepstral coefficient features and speech spectrogram features. The speech features are concatenated and input into the Transformer deep neural network model for streaming speech recognition to obtain dual-channel text sequences corresponding to the dual-channel audio. A unique identifier is generated based on channel recognition and speech energy difference, and a correspondence between the unique identifier and each text sequence is established. The unique identifier includes a role field and a timestamp field. The dialogue stream generation module is configured to use an LSTM-based model to predict punctuation and normalize text in a dual-channel text sequence, and sort and align it according to the timestamp field to generate a temporal dialogue stream corresponding to the dual-channel audio. An anomaly detection and storage module is configured to perform anomaly detection and / or storage management on time-series dialogue streams to enable real-time quality inspection, monitoring, and agent assistance.

[0021] According to a fourth aspect of the present invention, an application of a real-time dual-channel voice stream recognition method or system is proposed, which is applied to call quality inspection, agent assistance, and sensitive information monitoring scenarios in the banking, e-commerce, and government sectors. When applied to banking, e-commerce, and government sectors, this method or system can perform real-time recognition, structured storage, and anomaly monitoring of call content during call centers or voice interactions. Through simultaneous dual-channel voice recognition, timing alignment, and intelligent anomaly detection, it achieves automatic quality inspection of the call process, agent behavior-assisted decision-making, and sensitive information risk monitoring, thereby improving the quality of human service, reducing compliance risks, and enhancing business security.

[0022] This invention proposes a method, system, and application for real-time recognition of dual-channel speech streams, which has the following technical advantages: First, this invention achieves parallel recognition of customer and agent voices through dual-channel synchronous audio acquisition and time-series alignment processing. It also introduces an identity-assisted determination mechanism based on voice energy differences, enabling automatic differentiation of speaker roles during the recognition phase. This technical solution achieves an identity recognition accuracy rate of over 99.9%, completely resolving the problem of customer and agent information confusion in traditional single-channel voice recognition systems. Quality inspectors no longer need to perform manual track separation and labeling, significantly improving the automation level of quality inspection and data analysis.

[0023] Secondly, this invention employs a streaming Transformer model structure and a lightweight inference optimization strategy, enabling speech recognition, text transcription, and anomaly detection to form an end-to-end real-time processing link. The overall latency is controlled within 200ms (with recognition latency ≤100ms and detection and transcription latency ≤100ms), which is approximately 80% faster than existing offline processing modes (latency exceeding 1 second). This feature can meet the high real-time business requirements of call centers such as "real-time quality inspection" and "instant agent assistance," achieving instant response from call to analysis.

[0024] Furthermore, this invention proposes a hybrid detection mechanism based on the fusion of rule matching and deep learning models. It comprehensively utilizes the AC automaton algorithm, BERT semantic recognition, and LSTM attention sentiment analysis to achieve intelligent identification of multiple types of anomalies, including sensitive words, semantic violations, emotional abnormalities, and silence timeouts. Compared to traditional detection schemes that rely on keyword matching, this invention improves the anomaly identification accuracy to over 95% and reduces the false positive rate by approximately 60%, enabling more accurate reflection of risky behaviors and abnormal states during calls.

[0025] Furthermore, this invention, through business tag partitioning and distributed storage management mechanisms, can flexibly adapt to quality inspection, monitoring, and recording scenarios in different industries such as banking, e-commerce, and government. It enables rapid deployment without redundant development, reducing overall deployment and maintenance costs by approximately 40%. Simultaneously, employing AES-256 encryption, monthly key rotation, and off-site backup strategies, it ensures data security throughout the entire transmission and storage process, with a data leakage risk of less than 0.1%, meeting regulatory requirements such as the Personal Information Protection Law and the Financial Data Security Guidelines, and enabling traceability and recoverability of call data. Attached Figure Description

[0026] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many anticipated advantages of the embodiments will be readily recognized as they become better understood through reference to the following detailed description. Other features, objects, and advantages of this application will become more apparent from reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart of a real-time dual-channel voice stream recognition method according to an embodiment of this application; Figure 2 This is a flowchart of a real-time speech recognition embodiment of this application; Figure 3 This is a flowchart illustrating real-time transcription as an embodiment of this application; Figure 4 This is a flowchart of an anomaly detection method according to an embodiment of this application; Figure 5 This is a flowchart of a distributed storage system according to an embodiment of this application; Figure 6 This is a flowchart of a specific embodiment of the real-time dual-channel voice stream recognition application of this application; Figure 7 This is an architecture diagram of a dual-channel real-time speech stream recognition system according to a specific embodiment of this application; Figure 8 This is a schematic diagram of the structure of a computer system used to implement the electronic device of the present application. Detailed Implementation

[0027] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0028] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] Figure 1 A flowchart of a real-time dual-channel speech stream recognition method according to an embodiment of this application is shown. Figure 1 As shown, the method includes the following steps: S1: Preprocess the dual-channel audio of the acquired VoIP call and maintain the timing synchronization of the dual-channel audio during the preprocessing stage.

[0030] S2: Extract speech features from the preprocessed dual-channel audio, including Mel frequency cepstral coefficient features and speech spectrogram features. Concatenate the speech features and input them into the Transformer deep neural network model for streaming speech recognition to obtain dual-channel text sequences corresponding to the dual-channel audio. Generate unique identifiers based on channel recognition and speech energy difference, and establish the correspondence between the unique identifiers and each text sequence. The unique identifiers include role fields and timestamp fields.

[0031] In a specific embodiment, refer to Figure 2 , Figure 2 A flowchart illustrating a specific embodiment of real-time speech recognition according to this application is shown, such as... Figure 2 As shown, the real-time speech recognition steps include: Step 201: Receive dual-channel VoIP audio signals in PCM or WAV format, corresponding to the client and agent voice channels respectively. After reception, proceed to real-time preprocessing to ensure that the input signal amplitude and sampling rate meet the subsequent recognition requirements.

[0032] Step 202 involves audio preprocessing of the dual-channel audio signals, including three sub-steps: adaptive filtering for noise reduction, echo cancellation, and automatic gain control (AGC). Specifically, adaptive filtering uses the Least Mean Square Error (LMS) algorithm to iteratively optimize the filter weights, achieving dynamic suppression of background noise (such as office keyboard noise, air conditioner noise, etc.). The core logic of the LMS algorithm is to minimize the error signal... This is done to optimize the filter weights and minimize the noise component in the output signal. Specifically, the input signal is decomposed into a mixed signal. (Including target speech and background noise), the error signal is calculated as follows: ,in This represents the output noise of the filter. Filter weights. The formula is updated in real time by adjusting the error signal feedback. Among them, step size factor The step size is set to 0.02 to ensure appropriate weight update magnitude. In VoIP scenarios, this process can converge the error signal within 30-50 iterations, improving the signal-to-noise ratio by up to 15dB, significantly enhancing the clarity and recognizability of the speech signal. Through this noise reduction process, the filter effectively eliminates background noise, maximizing the preservation of the target speech and providing higher-quality speech input for subsequent feature extraction and speech recognition. Echo cancellation employs the Normalized Least Mean Square (NLMS) algorithm with a step size of... The filter order is 64 to suppress echo during calls in the 0-32ms range, with an echo suppression of approximately 20dB; Automatic gain control (AGC) stabilizes the signal amplitude at... Within a range of 20dB±3dB, prevent the input signal from being too weak (<-30dB) or too strong (>-10dB), which could cause signal overload and distortion.

[0033] Step 203 involves extracting features from the preprocessed audio signal to obtain multidimensional feature data that characterizes the speech spectrum and temporal properties; this step includes: Step 203a: Extract 13th-order MFCC features. Using a 25ms window length and a 10ms frame shift, extract 13th-order Mel-frequency cepstral coefficients (MFCC) features. Use 26 Mel filters, covering the 100Hz–8000Hz frequency band. The formula for calculating the Mel-frequency cepstral coefficients is as follows: In the formula, For the first The cepstral coefficients of the first Mel frequency (reflecting the speech frequency in the first Mel frequency) spectral envelope features on each cepstral dimension). For the first The output energy of a Mel filter (obtained by applying a Mel-scale filter bank to the speech spectrum); For the total number of Mel filters, preferably, The value is 26; The order of the cepstral coefficients is preferred. The range is 1 to 13. By taking the logarithm of the energy of the speech signal after passing through the Mel filter bank and applying a discrete cosine transform, a 13-dimensional cepstral feature vector is obtained, which is used to characterize the formants and envelope characteristics of the speech, thereby enhancing the recognition robustness under different speakers and noise environments.

[0034] Step 203b: Extract 128×128 spectrogram features. Short-Time Fourier Transform (STFT) is used to generate 128×128-dimensional speech spectrogram features, with the Hanning window as the window function. The features from steps 203a and 203b are concatenated at the frame level to form... A comprehensive feature sequence of dimensionality is used to enhance recognition robustness in noisy and dialectal environments.

[0035] Step 204: The concatenated feature sequence is input into a 12-layer Transformer deep neural network model optimized by ONNX for streaming speech recognition. The Transformer model outputs the corresponding speech-to-text sequence, achieving real-time conversion from acoustic features to text content; text segments are output synchronously based on timestamp information to ensure temporal alignment of the two-channel speech data. The Transformer deep neural network model achieves joint modeling of acoustic and linguistic features through a self-attention mechanism, and its attention calculation formula is as follows: In the formula, , , These are the query matrix, key matrix, and value matrix, each with dimensions of [batch size, sequence length, 512], where 512 represents the attention head feature dimension. The key matrix dimension is 512, used to normalize the similarity scores to prevent excessively large gradients from causing training instability. This is a matrix transpose operation. The calculation obtains relevance weights by performing a dot product on the query matrix and the key matrix, then normalizes them using Softmax and applies the weights to the value matrix. This enables global dependency modeling between features in each frame of the sequence, improving the context awareness and semantic consistency of speech recognition.

[0036] Furthermore, the training process for the Transformer deep neural network model is as follows: The training data consists of 500,000 hours of VoIP dual-channel speech corpus, covering Mandarin and 23 major dialects, and including various environmental noise conditions. The corpus is manually annotated with text content and speaker identification labels to construct a supervised learning sample set. The AdamW optimizer is used during the training phase, with the learning rate set to [value missing]. The weight decay coefficient is The batch size is 32, and the iterations are 200 rounds. The cross-entropy loss function is used as the objective function, and the model stabilizes when the loss value converges to below 0.05. During the inference phase, the trained Transformer model is exported to ONNX format for compression optimization, reducing the model size by approximately 40%. At the same time, the TensorRT engine is used to accelerate inference, achieving a single speech recognition latency of no more than 100ms, and compatibility with PCM / WAV audio input formats with sampling rates of 8kHz and 16kHz.

[0037] Step 205: Perform dual-channel labeling processing. Determine the current speaker's role based on channel identifiers and voice energy differences, and add a prefix of "customer" / "cust" or "agent" / "seat" to the recognition result. If the customer channel energy is detected to be approximately 5-8 dB lower than the agent channel, it is automatically labeled as a customer / cust speaking. Voice energy differences help distinguish identities and avoid misjudgments based solely on channel identifiers (e.g., customer voice interfering with the agent's channel when the agent is muted). Simultaneously, a unique identifier is generated for each recognition result in the format "role_ID_timestamp," where the role is "customer" / "cust" or "agent" / "seat," the ID is a random 8-character string, such as "a1b2c3d4," and the timestamp is accurate to milliseconds, such as "1620000000123." Step 206: Output the dual-channel recognition results in real time, package the structured text information into JSON format data, and push it at 50ms frame intervals via WebSocket streaming protocol. It supports automatic reconnection after disconnection and recovery mechanism for incomplete frames, ensuring a transmission success rate of ≥99.99%.

[0038] Example of outputting JSON format: [ {"speaker":"customer","content":"I forgot my bank card password","timestamp":1620000000123,"id":"cust_a1b2c3d4_1620000000123"}, {"speaker":"seat","content":"Please provide your ID number","timestamp":1620000002456,"id":"agent_e5f6g7h8_1620000002456"} ] S3: Utilize an LSTM-based model to predict punctuation and normalize text in dual-channel text sequences, and sort and align them according to the timestamp field to generate a temporal dialogue stream corresponding to the dual-channel audio.

[0039] In a specific embodiment, refer to Figure 3 , Figure 3 A flowchart illustrating real-time transcription of a specific embodiment of this application is shown, such as... Figure 3 As shown, the real-time transcription steps include: Step 301: Receive the transcription result in JSON format. The transcription result is output by real-time speech recognition and includes speaker role, speech content, and timestamp fields. The received transcription result is parsed and stored in a cache queue to provide input for subsequent format standardization and timing alignment processing.

[0040] Step 302, Timing Alignment Processing. Based on the timestamp field in each transcription result, the dual-channel speech text is quickly sorted using a fast sorting algorithm with a time complexity of O(n log n), ensuring that the customer and agent speech segments alternate in chronological order, such as "customer → agent → customer," forming a continuous interactive dialogue stream. Simultaneously, a unique call identifier (call ID) is generated for each dialogue stream. The call identifier format is "call_" + 16-bit UUID, enabling traceable binding of the original speech record, transcribed text, and anomaly detection results. Specifically, this includes: Step 1: Input Data Structure Definition. Each speech segment is defined as a standardized data structure (e.g., a JSON object) before sorting, containing the following fields: speaker: the speaker's identity (customer or agent); such as "customer".

[0041] content: Transcribed text content; such as "When will my order be shipped?".

[0042] start_ts: The start timestamp (in milliseconds) of the audio segment, which is used as the core basis for sorting; such as "1620000000123".

[0043] end_ts: The end timestamp of the audio segment (in milliseconds), used to help determine whether segments overlap; such as "1620000003456".

[0044] segment_id: A unique identifier for a segment; such as "cust_001".

[0045] Step 2, Quick Sort. First, select the reference value. Randomly select the start_ts of a segment from all voice segments as the reference value (pivot). For example, randomly select the segment with index k from segments, and the reference value pivot = segments[k].start_ts. This reference value is used to sort the voice segments by timestamp to avoid the worst-case scenario. Second, partition processing. Divide the voice segments into three parts: left, middle, and right: all segments with start_ts < pivot (segments earlier than the reference in time sequence); all segments with start_ts = pivot (a very special case where the timestamps are exactly the same); all segments with start_ts > pivot (segments later than the reference in time sequence). Implement the partition through the two-pointer method to ensure the accurate temporal position of the segments. Finally, recursively execute the same sorting process on the left and right partitions until all segments are sorted in ascending order of timestamp, forming a result that conforms to the dialogue alternation order. Among them, the partition process is achieved by traversing with two pointers: set the left pointer i (starting from 0) and the right pointer j (starting from len(segments)-1), and swap the segments that do not conform to the partition rule, finally making the left partition ≤ reference ≤ right partition.

[0046] Step 3, Exception Handling and Alternation Verification. Timestamp overlap handling: If the timestamp ranges of two segments overlap, preferentially retain the segment with the smaller start_ts, and mark the overlapping part as "[Overlap]". For example, if the customer segment [1620000000123, 1620000003456] overlaps with the agent segment [1620000002789, 1620000005123], then preferentially retain the segment with the smaller start_ts, retain the content of both in the overlapping part, and mark the "[Overlap]" prefix, for example, "Customer: My order [Overlap] Agent: Please wait a moment". Merge consecutive same-role segments: If there are consecutive segments of the same role after sorting, merge them into one segment, retain the earliest start_ts and the latest end_ts, and splice all the segment contents. Alternation verification: After sorting, check whether the dialogue flow conforms to the "Customer → Agent → Customer" alternation pattern; if it is found that more than two consecutive same-role segments are not merged, trigger a log alarm and mark it as "There may be a voice collection delay", but it does not affect the sorting result.

[0047] Step 4, Generate a unique call identifier for each dialogue flow, in the format of "call_" + 16-bit UUID (for example, "call_550e8400-e29b-41d4-a716-446655440000"), which is used to perform traceable binding of the original voice record, transcribed text, and anomaly detection results.

[0048] Step 303, Formatting Standardization. This includes two sub-steps: punctuation prediction and colloquial annotation. Punctuation prediction automatically adds periods, commas, and question marks to the transcribed text using a two-layer Long Short-Term Memory (LSTM) network model. When the model predicts a confidence probability greater than 0.7 for a certain type of punctuation, the punctuation is inserted at the corresponding position to restore semantic segmentation and syntactic integrity. The structure and training process of the LSTM model are as follows: The input layer uses 256-dimensional word embedding vectors, initialized based on pre-trained Sogou Chinese word vectors; the hidden layer consists of two LSTM layers, each containing 256 hidden units, with a dropout rate of 0.2 to prevent overfitting; the output layer is a four-class Softmax structure, corresponding to four output categories: no punctuation, comma, period, and question mark. The model training uses 100,000 manually annotated customer service dialogue texts as training data, achieving an annotation accuracy of 98%. The Adam optimizer is used during the training phase, with a learning rate set to 5×10. -5 The batch size is 64, the iteration is 100 rounds, the loss function is cross-entropy, the loss value is less than 0.1 when training converges, and the punctuation prediction accuracy on the validation set reaches 92%. During the inference phase, the model dynamically inserts punctuation based on the output probability. For example, when the probability of automatically predicting a period in "Customer: I forgot my bank card password" is greater than 0.7, the output is "Customer: I forgot my bank card password.", realizing automatic sentence segmentation and semantic recovery of real-time recognized text. In colloquial annotation, modal particles (such as "um", "ah", "this") are retained and annotated in the format of [modal particle] (such as "um [modal particle], I want to check the balance"). At the same time, the format of mixed Chinese and English text is optimized to ensure the consistency of the output results (such as "Your order No.12345 has been shipped").

[0049] Step 304, storage management operation. The transcribed results are written to a MongoDB 4.4 distributed database cluster after structuring. Hash sharding is performed using the call ID as the sharding key, and a single shard can store 5 million call records. To improve access performance, frequently accessed data from the last hour is also written to the Redis 6.2 cache with a TTL of 3600 seconds, accelerating hot data queries and reducing the query response time from 50ms in MongoDB to 1ms in Redis.

[0050] Step 305: Index Building Operation. A multi-dimensional inverted index is generated based on the Lucene 8.9 engine. The index dimensions include call ID, start and end timestamps, speaker identity, and transcribed text keywords. Keyword segmentation uses Jieba segmentation and supports custom dictionaries (such as "bank card password" and "overdue notification"). To balance real-time performance and retrieval efficiency, an "incremental indexing + periodic merging" strategy is adopted. An incremental index is generated every 10 minutes, and the entire index is merged at midnight every day. The construction time for a single index is less than 1ms. The system supports millisecond-level full-text search, with an average response time of less than 1ms.

[0051] S4: Perform anomaly detection and / or storage management on time-series dialogue streams to enable real-time quality inspection, monitoring, and agent assistance.

[0052] In a specific embodiment, refer to Figure 4 , Figure 4 A flowchart illustrating an anomaly detection of a specific embodiment of this application is shown, such as... Figure 4 As shown, the anomaly detection steps include: Step 410: Input the time-aligned dialogue text, which includes the call ID, speaker identity, and corresponding recognition text. Use this text as input to the anomaly detection engine for rule matching and model analysis.

[0053] Step 411: Load the dynamic rule base. The rule base is stored in JSON format and includes sensitive word rules (such as "password", "bank card number", "transfer limit", "refund", etc.) and service taboo phrases (such as "I don't know" "check it yourself"). It supports real-time updates via a web interface and automatically takes effect within 5 seconds of the update. To achieve efficient matching, this step uses the Aho-Corasick Automaton (AC automaton) algorithm to quickly match text with the rule base. Its core feature is that "all sensitive words can be matched in a single scan of the text", significantly improving matching efficiency in scenarios with multiple sensitive words.

[0054] Specifically, the construction process of the AC automaton includes three stages: rule base preprocessing, text matching, and performance optimization. First, in the rule base preprocessing stage, the sensitive word set is constructed into an automaton structure of "Trie + Failure Link + Output Function". Failure Links are generated using Breadth-First Search (BFS), allowing backtracking to the longest matching suffix node when a match fails, achieving backtrack-free scanning. The output function merges the sensitive word results of the current node and its failure link nodes to ensure that multiple levels of matching results can be output simultaneously when words are nested (e.g., "bank card" and "bank card number"). Second, in the text matching stage, the input text is scanned character by character starting from the root node. When a match fails, the system jumps to the failure link to continue matching. Each matched character checks the output list of the current node; if a sensitive word is found, the matching result is immediately recorded. For example, for the text "Customer: I forgot my bank card password", both "bank card" and "password" can be matched simultaneously in a single scan. The matching time for a single text is less than 1ms, and the matching complexity is O(log n). ,in, For text length, This represents the number of sensitive words matched. During the performance optimization phase, an architecture of "memory reuse + multi-threaded parallelism + double-buffer update mechanism" is adopted. Specifically, the same AC automaton instance is shared by multiple threads in read-only mode, with each thread independently processing a batch (1000 texts). Tasks are allocated through a task queue, enabling lock-free parallelism. Single-threaded matching speed can reach 2500 texts per second, and four threads in parallel can achieve batch matching of 10,000 texts per second. To support real-time updates to the sensitive word database, a new automaton copy is built in the background, and the primary and secondary buffers are switched atomically after the update is complete. The entire switching process takes less than 1ms, ensuring uninterrupted service.

[0055] Furthermore, the preprocessing of the rule base achieves efficient storage and retrieval of sensitive words by constructing a Trie structure of the AC automaton. The Trie serves as the basic data structure for sensitive words, with each node corresponding to a character, and the paths between nodes representing the prefix relationships of sensitive words. During construction, the root node is first initialized as the starting point for all sensitive words; then, each entry in the sensitive word database (such as "password" or "bank card number") is traversed sequentially, inserting it into Trie nodes character by character. The last character node is marked as the sensitive word end node, and its output list records the corresponding complete sensitive word. For example, inserting "password" forms the path "root→PIN→PIN", and the "PIN" node is marked with the output ["password"]; inserting "bank card number" forms the path "root→bank→card→number", and the "number" node is marked with the output ["bank card number"]. Finally, the path from the root node to any end node corresponds to a complete sensitive word.

[0056] Furthermore, to improve the matching efficiency, a failure pointer (Failure Link) for each node is constructed based on breadth-first search (BFS) to achieve non-backtracking jumps. When the matching fails, the algorithm can quickly backtrack to the longest suffix matching node along the failure pointer, thus avoiding re-scanning the text. The specific rules are as follows: The failure pointer of the root node is empty, and the failure pointers of its direct children point to the root node; for nodes with a depth greater than 1, let p be the failure pointer of its parent node. If p has a child node with the same character as the current character, the failure pointer of the current node points to that child node; otherwise, recursively search for the failure pointer of p until the root node. For example, for the child node "code" of the node "secret", if the root node does not have a suffix of "secret", the failure pointer of "code" points to the child node with the same name as the root node; if it still does not exist, it points to the root node itself.

[0057] Furthermore, to ensure the complete matching of nested vocabulary, the output function of the node needs to be merged with the output result of its failure pointer node. That is, the output list (output) of each node not only contains the sensitive words ending with itself, but also contains the output content of the node pointed to by its failure pointer. For example, when "bank card" is a sensitive word and "bank card number" is also a sensitive word, the output list of the node "number" needs to merge the output result of the node "card", that is, ["bank card number", "bank card"], so as to ensure that when "bank card number" is matched, nested words such as "bank card" can be recognized and output at the same time.

[0058] The Aho–Corasick automaton constructed through the above steps can complete the matching of all sensitive words at one time during the text scanning process, avoiding the performance loss caused by multiple regular expression matches, thus achieving efficient detection in the scenario of multiple sensitive words.

[0059] Step 412, determine whether the rule is hit. According to the matching result of the AC automaton, a quantitative evaluation is made on whether the detected text conforms to the sensitive word rule library, and a rule score is generated as the preliminary screening basis for the anomaly detection engine. The rule score is used to reflect the matching degree between the detected text and the rule library, and the value is 1, 0.5, or 0. The specific judgment logic is as follows: When any complete sensitive word in the rule library appears completely in the detected text, it is determined to be a complete hit, and the rule score is Ⅰ; when no complete word appears in the detected text, but it contains a valid fragment defined in the rule library, it is determined to be a partial hit, and the rule score is 0.5; when no sensitive word or valid fragment appears in the detected text, it is determined to be a miss, and the rule score is 0.

[0060] Furthermore, each sensitive word in the rule base contains two forms: a "complete word" and a "valid fragment." The valid fragment is not an arbitrary substring, but a core phrase with risk relevance, manually annotated by business personnel. For example, the complete word for the sensitive word "bank card password" is "bank card password," and the valid fragments are "bank card" and "password." The valid fragments for the sensitive word "transfer limit" are "transfer" and "limit." Words like "verification code" and "password," being short, have the same valid fragment as the complete word, and only score 1 point when they are a complete match. During text matching, the AC (Automatic Acquisition) system simultaneously identifies both complete words and valid fragments, achieving multi-layered matching and recognition. Taking a rule base containing "bank card password" and "transfer limit" as an example: When the detected text is "Please tell me your bank card password", it completely matches "bank card password", and the rule score is 1; when the detected text is "What is today's transfer limit?", it also completely matches, and the score is 1; when the detected text is "My bank card is lost", it only matches the fragment "bank card", not the complete word, and the score is 0.5; when the detected text is "This money needs to be transferred", it only matches the fragment "transfer", and the score is 0.5; if the detected text is "What procedures are needed for a return?", no sensitive words or fragments are matched, and the score is 0. When the same text matches multiple sensitive words simultaneously (such as "my bank card password and transfer limit"), the highest score is taken, and they are not accumulated. The definition of valid fragments in the rule base can be dynamically updated according to different industry scenarios. For example, in the financial quality inspection scenario, the combination of "bank card" and "password" is regarded as a high-risk feature. The rule base will be adjusted and supplemented with new high-risk phrases (such as the core fragment "digital currency" of "digital RMB") according to the quality inspection feedback.

[0061] Step 420, Model Analysis, includes three sub-steps: semantic violation detection, sentiment anomaly analysis, and silence detection. Semantic violation identification based on the BERT model is used to determine potential semantic violations in the dialogue content; sentiment anomaly analysis based on the LSTM attention mechanism model is used to identify statements containing emotions such as anger and dissatisfaction; silence detection is used to identify silence anomalies that exceed a preset duration during the call by calculating the time difference based on the timestamp fields of two adjacent texts.

[0062] BERT semantic violation detection is performed using a BERT-based-Chinese pre-trained model for fine-tuning. The input format is [CLS] + text + [SEP]. The model is trained on 50,000 manually annotated customer service dialogue texts, including 20,000 violation cases and 30,000 normal cases. The optimizer is AdamW (learning rate 2×10⁻⁶). -5 With a batch size of 16 and 50 iterations, the cross-entropy loss converged to below 0.05, and the F1 score for violation detection on the validation set reached 95%. The model outputs the violation probability of the text (0~1), such as "Can you tell me your bank card password?", which outputs 0.98.

[0063] The LSTM attention mechanism model is used to identify emotional tendencies and abnormal states in dialogue text. The model is built upon a Long Short-Term Memory (LSTM) network and an additive attention mechanism. Its input layer uses 256-dimensional word embedding vectors, initialized using pre-trained word vectors from Tencent AI Lab. The hidden layer is a single-layer LSTM structure containing 256 hidden units, with a dropout rate of 0.3 to prevent overfitting, effectively capturing the temporal emotional features of the text. The attention layer employs an additive attention mechanism, calculated using the following formula: In the formula, This is a trainable attention weight matrix; A trainable weight matrix (mapping matrix) used to represent the hidden state vectors. Projecting onto the attention space to extract latent emotional intensity features; For LSTM at time The output hidden state vector is weighted with emphasis on key sentiment words (such as "anger" and "dissatisfaction"). As a bias term, higher weights are assigned to words with emotional intensity (such as "anger" and "dissatisfaction") to achieve reinforcement learning of key emotional features. The output layer is a three-class Softmax structure, corresponding to the three emotional categories of "anger," "neutral," and "positive." The model training data includes 30,000 manually annotated customer service dialogue texts, of which 10,000 are in the anger category, 15,000 are in the neutral category, and 5,000 are in the positive category. The Adam optimizer is used during the training phase, with a learning rate of 5×10⁻⁶. -5 With a batch size of 32 and 80 iterations, the loss function is cross-entropy, and the final loss value converges to below 0.12. The sentiment recognition accuracy on the validation set reaches 93%. During the inference phase, when the probability of the "anger" emotion output by the model is greater than 0.6, it is determined that the text has an emotional abnormality and is marked as an abnormal event, thereby realizing real-time identification and early warning of abnormal emotions of customers or agents.

[0064] Silence detection is used to identify unnatural pauses during a call. It is an independent metric for determining temporal anomalies and is used to assess service fluency and interaction continuity. Its detection logic runs in parallel with the model score (used for sentiment and semantic violation judgment). Specifically, it is achieved by calculating the time difference between two adjacent valid texts (transcribed from the customer and agent). The formula for calculating the time difference is: In the formula, This is the start timestamp (start_ts) of the next text. This is the end timestamp (endts) of the previous text. When the timeout exceeds a preset threshold, it is considered an abnormal silence timeout. The default threshold is 30 seconds, which can be customized according to business scenarios. For example, in financial call scenarios, the threshold is set to 20 seconds to ensure customer response efficiency; in e-commerce customer service scenarios, the threshold is set to 30 seconds to accommodate the characteristics of consultation waiting. In addition, a "allow pause script" rule library is supported. When the previous text content contains preset pause scripts such as "Please wait, I'll check," the threshold is automatically temporarily relaxed to 60 seconds to avoid misjudging normal queries or system waiting as abnormal. After a silence timeout is detected, it is handled according to a three-level process of "real-time intervention - quality inspection traceability - data feedback." In the real-time intervention stage, a warning prompt is triggered 5 seconds in advance on the agent's end (such as "The current silence has exceeded 25 seconds, please respond to the customer in time"). If the timeout continues, a reassuring voice prompt is automatically played (such as "Transferring you to an agent, please wait"), and after exceeding the maximum threshold (e.g., 30 seconds in financial scenarios), the agent is automatically transferred to the shift leader's seat for manual intervention. During the quality inspection and traceability phase, the system labels the silent periods in the call text (e.g., "[Silence Timeout] 10:05:20–10:05:55 (35 seconds)") to help quality inspectors pinpoint the cause. Combined with logs, the source of the anomaly can be further differentiated: if it's due to an agent's prolonged lack of response, it's marked as "Service Process Violation"; if it's due to the customer's lack of response, it's marked as "Customer Reason"; if it's due to system delays, a technical investigation is automatically triggered. During the long-term feedback phase, by statistically analyzing the proportion and cause distribution of silence timeouts across various business scenarios, a "Silence Timeout Analysis Report" is automatically generated to optimize agent response standards and the voice acquisition module (e.g., adding caching strategies in weak network environments).

[0065] Furthermore, the silence detection result serves as an additional indicator for anomaly level determination, participating in the comprehensive risk assessment along with rule scores and model scores. When both "content risk" (e.g., model score ≥ 0.8) and "silence timeout" are present, the anomaly level is automatically upgraded to high risk; if only "silence timeout" is triggered without content risk, it is determined as a process anomaly event and used for service quality assessment. Through the above multi-layered detection and closed-loop processing mechanism, dual monitoring of "content compliance" and "interaction continuity" during the call process is achieved, effectively reducing customer churn and improving the overall service experience.

[0066] Step 421: Calculate the model score, which is the average of the BERT output violation probability and the sentiment abnormality probability.

[0067] Step 430: Combine the rule score and the model analysis score to calculate the combined score. The combined score formula is: Combined Score = Rule Score × 0.5 + Model Score × 0.5.

[0068] Step 431, determine the anomaly level. The anomaly level is determined based on the fusion score: When the fusion score < 0.5, it is determined as a general anomaly, such as "I want to inquire about the refund process," the rule was not hit, and the model's violation probability is 0.2; when 0.5 ≤ fusion score < 0.8, it is determined as a serious anomaly, such as "Can you help me check my transfer limit?" the rule hits a sensitive word, and the model's violation probability is 0.6; when the fusion score ≥ 0.8, it is determined as an urgent anomaly, such as "Tell me your bank card password to make a repayment," the rule hits a sensitive word, and the model's violation probability is 0.98. For general anomalies, logs are only recorded in the "anomaly_log" collection in the MongoDB database. The recorded fields include call ID, anomaly type, fusion score, and anomaly text content. For serious anomalies, alarm information is sent to quality inspectors via the cloud SMS API. The SMS template is "[Dual-channel voice system] Serious anomaly alarm: Call ID {call_id} contains sensitive content, fusion score {score}, please check immediately!". For emergency anomalies, alarm information is pushed to the agent supervisor's management interface in real time via WebSocket with a delay of ≤100ms, and the call is marked as "emergency intervention" to support immediate intervention by the supervisor.

[0069] Specifically, when the time difference of silence detection When a preset threshold is exceeded, the silent anomaly is incorporated into the anomaly detection logic as an independent temporal anomaly dimension, and a comprehensive evaluation is conducted through three steps: "temporal anomaly score quantification—score fusion correction—anomaly level re-determination." (Temporal anomaly score) This is used to quantify the degree of silence timeout in a call, and its value is positively correlated with the silence duration. Taking a financial scenario as an example, the threshold is set at 20 seconds; when the silence duration... When in different intervals, the corresponding score is calculated according to the gradient: when 20 seconds < When the time interval is ≤30 seconds, the timing anomaly score is 0.3, corresponding to a mild timing anomaly; when the time interval is <30 seconds... When the time interval is ≤60 seconds, the timing anomaly score is 0.6, corresponding to moderate timing anomaly; when For intervals exceeding 60 seconds, the timing anomaly score is 0.8, corresponding to severe timing anomalies. This classification can be adaptively adjusted based on the business scenario. For example, in e-commerce scenarios, where customer inquiry intervals are long, the system can... A time interval of one second corresponds to a temporal anomaly score of 0.2 to reduce the false positive rate. In the presence of silent anomalies, the fusion score calculation formula is expanded from the original two-dimensional calculation (rule score R and model score M) to a three-dimensional structure. The revised fusion score calculation formula is as follows: In the formula, the rule score is... The content matching risk based on the AC automaton is represented by the model score. This represents the temporal anomaly score for content risk prediction based on a deep learning model (BERT semantic violation probability and LSTM emotional anger probability). This indicates the delay risk at the call process level. The weighting logic is as follows: content risk dimension ( + The process anomaly dimension still accounts for the majority of the weight (0.8). It has an auxiliary weight (0.2) to ensure that the silence anomaly only plays a corrective role in the overall risk level, and does not dominate the risk assessment of the core content.

[0070] In a specific embodiment, if the rule score (Matching "password" number sense words), model score (Low semantic risk), when the silence time difference Lishi (corresponding to time series anomaly score) Substituting these values ​​into the correction formula, we get: Fusion Score = 1 × 0.4 + 0.1 × 0.4 + 0.3 × 0.2 = 0.5, corresponding to a severe anomaly level. When the silence time difference... Seconds (corresponding to severe timing anomalies) The calculation result is: Fusion Score = 1 × 0.4 + 0.1 × 0.4 + 0.8 × 0.2 = 0.6, still classified as a severe anomaly. This shows that silent anomalies have a corrective effect on the fusion result of "risk upgrading" rather than "level jump". Furthermore, to prevent process anomalies from offsetting content risks, a "level up without downgrading" strategy is adopted: when the original fusion score... When the severity level is severe or higher, even if the calculated result after adding the temporal anomaly score is slightly lower than the threshold, the original anomaly level remains unchanged; when the original fusion score is <0.5 (general anomaly) and the result after adding the temporal anomaly score is... Only when the time is right can the anomaly level be upgraded to a severe anomaly. Silent anomalies, as a "supplementary dimension to process risk," achieve a graded anomaly judgment mechanism that prioritizes content risk and supplements it with process risk through independent quantification and integrated correction. This design ensures comprehensive detection of process-related anomalies such as silent timeouts while preventing them from interfering with core content risk assessments, thus ensuring the rationality, stability, and business adaptability of anomaly judgments.

[0071] In a specific embodiment, refer to Figure 5 , Figure 5 A flowchart illustrating a specific embodiment of distributed storage of this application is shown, such as... Figure 5 As shown, the distributed storage steps include: Step 510: Receive the time-aligned call text and recording files, and write the data to the distributed storage cluster. The storage cluster consists of three or more physical nodes. Each node's hardware configuration includes a CPU with 16 or more cores, at least 64GB of RAM, and a 2TB solid-state drive (SSD). Load balancing is achieved based on a consistent hashing algorithm. Each physical node corresponds to 100 virtual nodes to ensure even data distribution across the cluster, with load deviation between nodes controlled within 10%. To improve reliability and disaster recovery capabilities, the cluster adopts a "one master, two slaves" replication synchronization strategy. The master node is responsible for data writing, and the slave nodes perform real-time data mirroring synchronization. When the master node fails, the slave nodes automatically elect a new master node through the Raft consensus protocol. The election and switching process takes no more than 300ms, thus ensuring the continuity and high availability of data services, with overall data availability reaching 99.99%.

[0072] Step 520, Secure Encryption Processing. Received call text and recording data are encrypted using the AES-256 symmetric encryption algorithm. The key is generated and managed by a Key Management System (KMS), and the master key is automatically rotated monthly to prevent security risks from prolonged use. During data encryption, the original voice text (e.g., "Customer: My bank card number is 6222XXXX") is encrypted with the master key to generate ciphertext (e.g., "U2FsdGVkX1+..."). The encrypted master key is stored in a Hardware Security Module (HSM) to prevent key leakage or malicious access. During data transmission, a multi-layered encryption mechanism is employed to ensure communication security: voice signaling (including SIP REGISTER, INVITE, etc.) is encrypted using the TLS 1.3 protocol to prevent interception of call signaling at the network layer; application interface (API) access and log export processes use the HTTPS encryption protocol and are loaded with an Extended Validation Digital Certificate (EV SSL) to ensure the authenticity and integrity of the transmitted content. The aforementioned dual-layer encryption mechanism effectively prevents call data from being stolen, tampered with, or leaked during transmission and storage, significantly improving the system's data security and compliance.

[0073] Step 530, Partition Management. Hash partitioning is performed based on "Date + Business Tag," with the partition format "YYYY-MM-DD_Business Tag" (e.g., "2024-05-01_Banking Business" "2024-05-01_E-commerce Customer Service"), to achieve logical grouping and independent management of data. When users query data by business type, only the corresponding partition is retrieved, reducing the query scope by 80% and the single query response time from 50ms to 10ms, significantly improving retrieval efficiency.

[0074] Step 504, Lifecycle Management. Regularly check data storage duration and implement a tiered management strategy. When storage time exceeds one year, automatically migrate the data to a tape library for archiving. Before migration, calculate the MD5 checksum to verify data integrity after migration. For financial compliant call data (such as bank collections, overdue notifications, etc.), automatically perform off-site backups at 3 AM daily. The backup location is approximately 300km from the main center, and the data retention period is 7 years to meet the regulatory requirements of the "Financial Data Security Guidelines." For non-compliant data stored for more than 7 years (such as ordinary e-commerce customer service call records), the system automatically performs a cleanup task. Before cleanup, notify the administrator via email for confirmation, and then perform the deletion operation to release storage resources and ensure long-term stable system operation.

[0075] Example Taking a real-time quality inspection scenario in a bank's VoIP call center as an example, the deployment environment consists of both hardware and software components. On the hardware side, there are two voice processing servers, configured with Intel Xeon Gold 6330 CPUs, 64GB of RAM, and 2TB SSDs, used to run the real-time voice recognition and anomaly detection engine; and three storage servers, configured with Intel Xeon Silver 4314 CPUs, 128GB of RAM, and 10TB SSDs, used to deploy a MongoDB and Redis cluster; the dual-channel synchronous audio capture card uses an RMEHammerfall DSP, supporting 8kHz and 16kHz sampling rates with a synchronization accuracy of less than 1ms. On the software side, the system runs on CentOS 7.9, with audio signal format of 16-bit PCM at a sampling rate of 16kHz. The configuration includes a Transformer recognition model fine-tuned with 100,000 hours of bank customer service voice recordings, a BERT classification model fine-tuned for financial violation semantic scenarios, and an LSTM sentiment model trained on bank customer service sentiment data. Data storage utilizes a MongoDB 4.4 three-shard cluster, a Redis 6.2 three-master three-slave cluster, and an IBM TS4500 tape library. The anomaly detection rule base contains approximately 500 financially sensitive words (such as "password," "bank card number," and "transfer"), with a silence timeout threshold of 20 seconds. The fusion score thresholds are divided into general <0.5, severe 0.5–0.8, and urgent ≥0.8.

[0076] refer to Figure 6 , Figure 6 A flowchart illustrating a specific embodiment of the real-time dual-channel speech stream recognition application of this application is shown. Figure 6 As shown, the implementation process includes the following steps: Step 601, Dual-channel voice input. The RME Hammerfall DSP audio acquisition card synchronously receives VoIP voice signals from the client (channel 1) and the agent (channel 2), using the UDP protocol for streaming transmission with a transmission delay of less than 20ms; the voice processing server enables dual threads to receive the two audio channels separately, ensuring signal independence and preventing crosstalk.

[0077] Step 602, real-time speech recognition. This specifically includes the following: a. Audio preprocessing. LMS adaptive filtering is used to remove background noise (such as bank lobby queuing sounds), improving the customer's voice signal-to-noise ratio from 10dB to 25dB; the NLMS algorithm is used to suppress headphone leakage, with an echo suppression of approximately 20dB; automatic gain control (AGC) stabilizes the audio amplitude at... Within a range of 20dB±3dB, avoid signal weakness that could lead to recognition loss.

[0078] b. Feature Extraction. 13th-order MFCC features (Mel filter number M=26) and 128×128 spectral features are extracted, concatenated, and input into the Transformer model for streaming speech recognition. The Transformer model recognizes the customer's speech as "I forgot my bank card password" (80ms delay) and the agent's speech as "Please provide your ID number" (75ms delay).

[0079] c. Dual-path tagging. The recognition output is encapsulated in JSON format, including role identifier, timestamp, and unique ID field (such as "cust_a1b2c3d4_1620000000123"), and is transcribed in real time via WebSocket streaming.

[0080] Step 603: Real-time transcription and storage.

[0081] The system automatically adds punctuation marks based on a two-layer LSTM punctuation prediction model, transforming the recognized text into "Customer: I forgot my bank card password." and "Agent: Please provide your ID number." Since this scenario does not contain colloquialisms, no additional annotation is required.

[0082] Based on the timestamp fields (customer 1620000000123, agent 1620000002456), sort the two-way text, generate the interaction stream "customer → agent", and bind a unique call ID "call_550e8400-e29b-41d4-a716-446655440000".

[0083] The time-series text is written to the MongoDB collection "bank_call" (shard key is call ID) and simultaneously to the Redis cache (TTL=3600 seconds); the Lucene index engine builds keyword indexes ("bank card password", "ID number"), time indexes ("2024-05-01"), and business tags ("banking business") on the text to support subsequent retrieval and audit tracing.

[0084] Step 604a, Anomaly Detection. The AC automaton hits the sensitive word "password", and the rule score is recorded as 1; the BERT semantic model judges the text semantics as "normal password forgetting consultation", with a violation probability of 0.2; the LSTM attention sentiment model identifies the sentiment as "neutral", with an anger probability of 0.1; the silence detection module calculates the time difference of 2.33 seconds, which is less than the threshold of 20 seconds, so there is no silence anomaly, and the model score is (0.2+0.1) / 2=0.1; level determination: fusion score = 1×0.5+0.1×0.5=0.55, which is judged as "serious anomaly"; an alarm message is sent to the quality inspection personnel via SMS API, and the SMS content is "[Dual-channel voice system] Serious anomaly alarm: Call ID call_550e8400-e29b-41d4-a716-446655440000 contains sensitive content, fusion score 0.55, please check in time!".

[0085] Step 604b, Distributed Storage. For quality inspection and data management, quality inspectors retrieve complete conversation records and anomaly logs through the call ID in the quality inspection management interface, verifying whether the agent's response complies with banking service standards (confirming no leakage of customer information and correct guidance for identity verification), and marking the quality inspection result as "qualified." Data lifecycle management is implemented: call data is automatically migrated to the IBM TS4500 tape library after one year of storage. An MD5 digest is generated and verified before migration, ensuring 100% data integrity. Automatic off-site backup is performed daily at 3 AM, synchronizing the backup data to the Shanghai disaster recovery center. The retention period is 7 years, complying with financial industry compliance requirements.

[0086] Further reference Figure 7 As an implementation of the above method, this application provides an embodiment of a dual-channel voice stream real-time recognition system 700, which is similar to... Figure 1 Corresponding to the illustrated method embodiment, this system can be specifically applied to various electronic devices. The system 700 includes an audio preprocessing module 701, a speech recognition module 702, a dialogue stream generation module 703, and an anomaly detection and storage module 704, which are communicatively connected to each other, wherein: The audio preprocessing module 701 is configured to preprocess the dual-channel audio of the acquired VoIP call and maintain the timing synchronization of the dual-channel audio during the preprocessing stage. The speech recognition module 702 is configured to extract speech features from preprocessed dual-channel audio. The speech features include Mel-frequency cepstral coefficient features and speech spectrogram features. The speech features are concatenated and input into the Transformer deep neural network model for streaming speech recognition to obtain dual-channel text sequences corresponding to the dual-channel audio. A unique identifier is generated based on channel recognition and speech energy difference, and a correspondence between the unique identifier and each text sequence is established. The unique identifier includes a role field and a timestamp field. The dialogue stream generation module 703 is configured to use an LSTM-based model to predict punctuation and normalize text in a dual-channel text sequence, and to sort and align the text according to the timestamp field to generate a temporal dialogue stream corresponding to the dual-channel audio. The anomaly detection and storage module 704 is configured to perform anomaly detection and / or storage management on the time-series dialogue stream to achieve real-time quality inspection, monitoring and agent assistance.

[0087] like Figure 8 As shown, the computer system includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 802 or programs loaded from storage section 808 into random access memory (RAM) 803. RAM 803 also stores various programs and data required for the operation of system 800. CPU 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0088] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a liquid crystal display (LCD) and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card and a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed.

[0089] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by central processing unit (CPU) 801, it performs the functions defined in the methods of this application. It should be noted that the computer-readable storage medium of this application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0090] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0091] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0092] The modules described in the embodiments of this application can be implemented in software or in hardware.

[0093] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: preprocess the acquired dual-channel audio of a VoIP call, maintaining temporal synchronization of the dual-channel audio during the preprocessing stage; extract speech features from the preprocessed dual-channel audio, including Mel-frequency cepstral coefficient features and speech spectrogram features, concatenating the speech features and inputting them into a Transformer deep neural network model for streaming speech recognition to obtain dual-channel text sequences corresponding to the dual-channel audio; generate unique identifiers based on channel recognition and speech energy difference, and establish a correspondence between the unique identifiers and each text sequence, wherein the unique identifiers include a role field and a timestamp field; perform punctuation prediction and text normalization on the dual-channel text sequences using an LSTM-based model, and sort and align them according to the timestamp field to generate a temporal dialogue stream corresponding to the dual-channel audio; and perform anomaly detection and / or storage management on the temporal dialogue stream.

[0094] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for real-time recognition of dual-channel speech streams, characterized in that, include: S1: Preprocess the dual-channel audio of the acquired VoIP call, and maintain the timing synchronization of the dual-channel audio during the preprocessing stage; S2: Extract speech features from the preprocessed dual-channel audio, including Mel-frequency cepstral coefficient features and speech spectrogram features. Concatenate the speech features and input them into the Transformer deep neural network model for streaming speech recognition to obtain the dual-channel text sequence corresponding to the dual-channel audio. Generate a unique identifier based on channel recognition and speech energy difference, and establish the correspondence between the unique identifier and each text sequence. The unique identifier includes a role field and a timestamp field. S3: Use an LSTM-based model to predict punctuation and normalize the text in the dual-channel text sequence, and sort and align it according to the timestamp field to generate the temporal dialogue stream corresponding to the dual-channel audio. S4: Perform anomaly detection and / or storage management on the time-series dialogue stream to achieve real-time quality inspection, monitoring and agent assistance.

2. The real-time dual-channel speech stream recognition method according to claim 1, characterized in that, S4 includes a fusion detection mechanism based on rule matching and model analysis to detect anomalies in the temporal dialogue stream. The fusion detection mechanism determines the anomaly level based on the rule score obtained from rule matching and the model score obtained from model analysis, and triggers a corresponding response result based on the anomaly level. The rule matching is based on the AC automaton algorithm to perform multi-pattern matching on the text to detect sensitive words and service taboo phrases. The model analysis includes semantic violation recognition based on the BERT model and sentiment anomaly analysis based on the LSTM attention mechanism model.

3. The real-time dual-channel speech stream recognition method according to claim 2, characterized in that, The fusion detection mechanism also includes a timing anomaly determination based on the timestamps of two adjacent texts, calculating the time difference to detect silence timeout anomalies during calls; the time difference calculation formula is: In the formula, This is the start timestamp of the next text. The end timestamp of the previous text; in response to the time difference. If the time exceeds a preset threshold, it is determined to be a time-series anomaly. The time-series anomaly score is quantified based on the silence duration and combined with the rule score and the model score to re-determine the anomaly level.

4. The real-time dual-channel speech stream recognition method according to claim 3, characterized in that, The LSTM attention mechanism model is constructed based on an LSTM network and an additive attention mechanism. The LSTM network is used to extract temporal features from the speech-to-text transcription to capture the emotional dependencies of the contextual semantics. The calculation formula for the additive attention mechanism is as follows: In the formula, For a trainable attention weight matrix, A trainable weight matrix is ​​used to train the hidden state vector. Projecting onto the attention space to extract latent emotional intensity features. The bias term is used; the output layer of the LSTM attention mechanism model adopts a Softmax classification structure to classify text into three emotion categories: angry, neutral, and positive.

5. The real-time dual-channel speech stream recognition method according to claim 1, characterized in that, The S4 further includes performing AES-256 symmetric encryption on the time-series dialogue stream and then distributing it for storage. The distributed storage is based on a consistent hashing algorithm to achieve data load balancing, adopts a "one master, two slaves" replica synchronization strategy for redundant data storage, and performs partition management and lifecycle control based on date and business tags.

6. The real-time dual-channel speech stream recognition method according to claim 1, characterized in that, The preprocessing includes adaptive filtering noise reduction, echo cancellation, and automatic gain control. The adaptive filtering noise reduction uses the least mean square algorithm, and the echo cancellation uses the normalized least mean square algorithm.

7. The real-time dual-channel speech stream recognition method according to claim 1, characterized in that, The Mel frequency cepstral coefficient features are extracted using a 25ms window and a 10ms frame shift to obtain 13th-order Mel frequency cepstral coefficient features. The number of Mel filters is 26, covering the frequency band from 100Hz to 8000Hz. The formula for calculating the Mel frequency cepstral coefficients is as follows: In the formula, For the first Mehr frequency cepstral coefficients, For the first The output energy of a Mel filter The total number of Mel filters; The order of the cepstral coefficients; the speech spectrogram features are generated by short-time Fourier transform, and the spectrogram dimension is 128×128.

8. The real-time dual-channel speech stream recognition method according to claim 1, characterized in that, S3 further includes building a multi-dimensional inverted index for the generated time-series dialogue stream. The index dimensions of the inverted index include call identifier, timestamp field, role field, and transcribed text keywords. The multi-dimensional inverted index construction process adopts an incremental update and periodic merging strategy to balance real-time data writing and full-text retrieval efficiency.

9. The real-time dual-channel speech stream recognition method according to claim 1, characterized in that, The attention calculation formula for the Transformer deep neural network model is as follows: In the formula, , , These are the query matrix, key matrix, and value matrix, each with dimensions of [batch size, sequence length, 512]. The dimension of the key matrix is... This is the matrix transpose operation.

10. A real-time dual-channel speech stream recognition system, characterized in that, include: An audio preprocessing module is configured to preprocess the dual-channel audio of the acquired VoIP call and maintain the timing synchronization of the dual-channel audio during the preprocessing stage. The speech recognition module is configured to extract speech features from preprocessed dual-channel audio. The speech features include Mel-frequency cepstral coefficient features and speech spectrogram features. The speech features are concatenated and input into a Transformer deep neural network model for streaming speech recognition to obtain dual-channel text sequences corresponding to the dual-channel audio. A unique identifier is generated based on channel recognition and speech energy difference, and a correspondence between the unique identifier and each text sequence is established. The unique identifier includes a role field and a timestamp field. The dialogue stream generation module is configured to use an LSTM-based model to predict punctuation and normalize the text sequence of the two-channel audio, and to sort and align the sequences according to the timestamp field to generate a temporal dialogue stream corresponding to the two-channel audio. An anomaly detection and storage module is configured to perform anomaly detection and / or storage management on the time-series dialog stream.

Images