Dialogue abnormity real-time detection method and system based on multi-dimensional feature fusion

Abstract

The invention relates to a dialogue abnormity real-time detection method and system based on multi-dimensional feature fusion. The method comprises the steps that a streaming text generated by ASR and acoustic feature data of user voice are received in real time; performing behavior abstraction on the AI reply to form a tag sequence, and matching rigid repetition, invalid cycle and other structural anomalies in real time through an abnormal mode rule base; calculating a negative emotion score through a multi-mode emotion model in combination with the user text and the acoustic features, and calculating a semantic correlation score of sentences of two dialogue parties; and finally, performing comprehensive judgment by fusing the behavior anomaly, the emotion score and the semantic correlation three-dimensional signal, and generating a structured anomaly mark when a condition is met. According to the method, the limitation of traditional off-line quality inspection is broken through, real-time detection is achieved, semantic relation and context logic abnormity which cannot be found by a traditional method are effectively recognized through multi-modal fusion analysis, and the accuracy and automation level of dialogue quality monitoring are remarkably improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and system for real-time detection of dialogue anomalies based on multi-dimensional feature fusion. Background Technology

[0002] With the rapid development of large language model (LLM) technology, conversational AI systems based on automatic speech recognition (ASR), natural language processing (NLP), and text-to-speech (TTS) technologies have been widely applied in business scenarios such as telemarketing and customer service. The conversational quality of these systems directly affects user experience and business performance.

[0003] Currently, the industry primarily relies on offline, single-dimensional analysis methods for detecting such interaction anomalies. One representative existing technology is an offline text quality inspection system based on keyword matching and sentiment analysis. This approach transcribes the recording into text after the call ends, then scans and scores the call content using a pre-built negative keyword dictionary or a text-based single-modal sentiment analysis model. Finally, human review of the problematic calls marked by the system is conducted. This method forms the mainstream technological foundation for current dialogue quality assessment.

[0004] The aforementioned technology suffers from severe lag, with all analysis conducted after the call ends, making it impossible to intervene or provide real-time feedback. Furthermore, its analytical dimensions are extremely limited, relying entirely on ASR-transcribed text and completely discarding key acoustic information such as tone, intonation, and speech rate contained in the voice, which can easily lead to misjudgment of the user's true emotions.

[0005] Based on this, this application provides a method and system for real-time detection of dialogue anomalies based on multi-dimensional feature fusion. Summary of the Invention

[0006] To address the issues of severe lag, where all analysis is performed after the call ends, making in-process intervention and real-time feedback impossible; secondly, its analysis dimensions are extremely limited, relying entirely on ASR transcription text and completely discarding key acoustic information such as tone, intonation, and speech rate contained in the speech, which can easily lead to misjudgment of the user's true emotions, this application provides a real-time dialogue anomaly detection method and system based on multi-dimensional feature fusion.

[0007] Firstly, this application provides a real-time dialogue anomaly detection method based on multi-dimensional feature fusion, employing the following technical solution: including: Receive and recognize streaming text data generated from the user's original speech, and extract acoustic feature data from the user's original speech in real time; Based on the responses generated by the dialogue module in the streaming text data, behavior abstraction is performed to form a behavior tag sequence, and the behavior tag sequence is matched in real time based on a preset abnormal pattern rule base to obtain the abnormal detection results of the behavior pattern. Based on the user speech text in the streaming text data and the time-aligned acoustic feature data, a negative emotion score is calculated and output through a preset multimodal emotion classification model; at the same time, the similarity between the text embedding vector of the current response of the dialogue module and the user's previous speech is calculated to obtain a semantic relevance score. The anomaly detection results, the negative emotion score, and the semantic relevance score are integrated and comprehensively evaluated according to a preset fusion judgment logic. When the anomaly triggering condition is met, an anomaly marker is generated.

[0008] Preferably, the step of receiving and recognizing streaming text data generated from the user's original speech, and extracting acoustic feature data from the user's original speech in real time, includes: The system uses an automatic speech recognition module to recognize the user's original speech in real time, generating streaming text data organized according to dialogue turns; and... The acoustic feature data is extracted from the user's original speech in real time through an independent audio processing module and an automatic speech recognition module. The acoustic feature data includes, but is not limited to, one or more of pitch, energy, and speech rate and their statistical features.

[0009] Preferably, the step of performing behavior abstraction based on the responses generated by the dialogue module in the streaming text data to form a behavior tag sequence, and performing real-time matching of the behavior tag sequence based on a preset abnormal pattern rule base to obtain abnormal detection results of the behavior pattern includes: By using preset mapping rules, each round of response text generated by the dialogue module is mapped to a predefined behavior label, and the behavior label sequence is generated by arranging them in chronological order. The mapping rules are implemented based on keyword matching, intent recognition models, or regular expressions. Within a real-time sliding dialog window, the sequence of behavioral tags is matched with predefined abnormal interaction patterns in the abnormal pattern rule base, wherein the abnormal interaction patterns include at least rigid repetition patterns and invalid loop patterns. The rigid repetition mode is defined as: within a preset number of consecutive dialogue rounds, the frequency of occurrence of the same behavior tag exceeds a first threshold; the invalid loop mode is defined as: within a preset number of consecutive dialogue rounds, the number of times a specific behavior tag subsequence is repeated exceeds a second threshold.

[0010] Preferably, before generating an anomaly marker by fusing the anomaly detection result, the negative sentiment score, and the semantic relevance score, and performing a comprehensive evaluation according to a preset fusion judgment logic, the process further includes: The rules and thresholds in the abnormal pattern rule base are dynamically configured through statistical analysis of historical abnormal dialogue data. The statistical analysis includes: calculating the correlation between the frequency of occurrence of specific behavior tags or behavior sequences in historical dialogues and the final dialogue failure rate; and adaptively adjusting the first threshold for triggering the rigid repetition pattern or the second threshold for triggering the invalid loop pattern based on the correlation analysis results.

[0011] Preferably, the acoustic feature data based on user speech text and time alignment in the streaming text data is used to calculate and output a negative emotion score through a preset multimodal emotion classification model; simultaneously, the similarity between the text embedding vector of the current response of the dialogue module and the user's previous speech is calculated to obtain a semantic relevance score, including: The time-aligned user speech text and the acoustic feature data are jointly input into a pre-trained multimodal emotion classification model. The multimodal emotion classification model performs joint inference to obtain the negative emotion score, which is a continuously valued representation of the user's current negative emotion level. The pre-trained language model is used to generate text embedding vectors for the user's previous utterance and the current response from the dialogue module, and the semantic relevance score is obtained by calculating the cosine similarity between the two text embedding vectors.

[0012] Preferably, before inputting the time-aligned user speech text and the acoustic feature data into a pre-trained multimodal emotion classification model, the method further includes: Training the multimodal emotion classification model: Obtain a training dataset containing a large number of dialogue samples labeled with negative emotion tags. Each sample contains user speech text, time-aligned acoustic feature data, and corresponding real emotion tags. A multimodal fusion network is constructed based on deep learning. The multimodal fusion network includes at least a text encoder for processing text features, an acoustic encoder for processing acoustic features, and a fusion classifier for fusing bimodal information. Supervised learning is used to train the multimodal fusion network end-to-end with the goal of minimizing the difference between the model's predicted results and the actual sentiment labels, until the model converges.

[0013] Preferably, the fusion of the anomaly detection result, the negative sentiment score, and the semantic relevance score is comprehensively evaluated according to a preset fusion judgment logic. When the anomaly triggering condition is met, an anomaly marker is generated, including: When the anomaly detection result indicates the presence of an anomaly, or when the negative emotion score exceeds a preset score threshold, or when the semantic relevance score is lower than a preset similarity threshold, the anomaly triggering condition is determined to be met, and the anomaly marker is generated. The anomaly marker is a structured data object, which includes at least the anomaly triggering time, anomaly level, the type of anomaly triggered, and the associated dialogue context text that triggered the anomaly.

[0014] Secondly, this application discloses a real-time dialogue anomaly detection device based on multi-dimensional feature fusion, which adopts the following technical solution, including: The speech recognition module is used to receive and recognize streaming text data generated from the user's original speech, and to extract acoustic feature data from the user's original speech in real time. The behavior determination module is used to perform behavior abstraction to form a behavior tag sequence based on the responses generated by the dialogue module in the streaming text data, and to perform real-time matching of the behavior tag sequence based on a preset abnormal pattern rule base to obtain the abnormal detection results of the behavior pattern. The semantic analysis module is used to calculate and output a negative emotion score based on the user's speech text in the streaming text data and the time-aligned acoustic feature data, through a preset multimodal emotion classification model; at the same time, it calculates the similarity between the text embedding vector of the current response of the dialogue module and the user's previous speech to obtain a semantic relevance score. The anomaly labeling module is used to integrate the anomaly detection results, the negative emotion score, and the semantic relevance score, and to perform a comprehensive evaluation based on a preset fusion judgment logic. When the anomaly triggering condition is met, an anomaly label is generated.

[0015] Thirdly, this application also provides a control device, the device comprising: It includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed, such as the real-time dialogue anomaly detection method based on multi-dimensional feature fusion described above.

[0016] Fourthly, this application also provides a computer-readable storage medium storing a computer program that can be loaded by a processor and executed as described above in the real-time dialogue anomaly detection method based on multi-dimensional feature fusion.

[0017] In summary, during the dialogue process, the system receives in parallel a multimodal data stream consisting of streaming text generated by ASR and acoustic features extracted from the user's speech. This data stream is then simultaneously fed into two analysis channels: the first is behavioral pattern analysis, which abstracts AI responses into predefined behavioral labels and forms sequences, using an anomaly pattern rule base for real-time matching to identify structural anomalies such as rigid repetition and invalid loops; the second is joint emotion and semantic analysis, which calculates negative emotion scores by fusing text and acoustic features through a multimodal emotion classification model, and simultaneously obtains semantic relevance scores by calculating the cosine similarity of the embedding vectors of the two parties' statements. Finally, the system integrates the aforementioned behavioral anomaly signals, emotion scores, and semantic relevance scores, and makes a comprehensive decision based on preset logic. Once the conditions are met, a structured anomaly marker is generated. By using multimodal parallel processing and feature fusion, real-time detection of interaction anomalies is achieved, overcoming the lag of traditional offline quality inspection. At the same time, by introducing the analysis of user voice acoustic features, the shortcomings of pure text analysis are made up for, and the true emotions conveyed by users through tone of voice and other means can be captured, making emotion judgment more accurate, thereby significantly improving the automation level and depth of dialogue monitoring. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a real-time dialogue anomaly detection method based on multi-dimensional feature fusion.

[0019] Figure 2 This is a structural block diagram of a real-time dialogue anomaly detection device based on multi-dimensional feature fusion. Detailed Implementation

[0020] The following combination Figure 1 - Figure 2 This application will be described in further detail.

[0021] Currently, the monitoring and detection of these abnormal conversations mainly rely on the following methods: Offline manual sampling inspection: After the call ends, quality inspectors randomly select recordings for playback and evaluation. This method is costly, inefficient, has severely insufficient coverage, and causes significant delays in problem detection, making it impossible to intervene in the online system in real time or iterate rapidly.

[0022] Keyword-based rule matching identifies problematic calls by searching for negative keywords (such as "can't hear clearly," "wrong," "complaint") in the call text. While simple, this method has limited accuracy, failing to capture a large amount of implicit, nonverbal user dissatisfaction (such as impatient tone or prolonged silence) and failing to identify structural problems in the conversation flow itself.

[0023] Single-dimensional scoring models: Some systems score calls in a single dimension (such as satisfaction prediction), but this is usually a comprehensive post-event evaluation that cannot pinpoint specific abnormal interaction segments in real time during the call, nor can it reveal the root cause of the abnormality.

[0024] Based on this, the main purpose of this application is to overcome the above-mentioned defects of the prior art and provide a real-time detection method and system for dialogue anomalies based on multi-dimensional feature fusion, which aims to achieve real-time, accurate and automated detection of interaction anomalies in human-computer voice dialogue.

[0025] Reference Figure 1 The embodiments of this application include at least steps S10 to S40.

[0026] S10 receives and recognizes streaming text data generated from the user's original speech, and extracts acoustic feature data from the user's original speech in real time.

[0027] S20: Based on the responses generated by the dialogue module in the streaming text data, perform behavior abstraction to form a behavior label sequence, and perform real-time matching of the behavior label sequence based on the preset abnormal pattern rule base to obtain the abnormal detection results of the behavior pattern.

[0028] S30: Based on user speech text and time-aligned acoustic feature data in streaming text data, a negative emotion score is calculated and output through a pre-set multimodal emotion classification model; at the same time, the similarity between the text embedding vector of the current response of the dialogue module and the user's previous speech is calculated to obtain a semantic relevance score.

[0029] S40 integrates anomaly detection results, negative sentiment scores, and semantic relevance scores, and performs a comprehensive evaluation based on a preset fusion judgment logic. When the anomaly triggering conditions are met, an anomaly marker is generated.

[0030] Specifically, the system receives streaming text converted from ASR and acoustic features extracted from the original speech in parallel to build a multimodal data foundation. Then, it abstracts the AI ​​responses into a label sequence and identifies structural anomalies such as rigid repetitions and invalid loops by matching them with a preset rule base in real time. At the same time, it jointly analyzes the user's text and corresponding acoustic features, calculates negative emotion scores through a multimodal model, and uses embedded vectors to calculate semantic relevance to determine irrelevant answers. Finally, it integrates the three-dimensional signals of behavioral anomalies, emotion scores, and semantic relevance, and makes a comprehensive judgment based on preset logic. When the conditions are met, it immediately generates a structured anomaly label, thereby achieving real-time, multi-dimensional, and accurate capture of interaction anomalies.

[0031] In some embodiments, step S10 specifically includes the following steps: performing real-time recognition of the user's original speech through an automatic speech recognition module to generate streaming text data organized by dialogue turns; and extracting acoustic feature data from the user's original speech in real-time through an independent audio processing module in parallel with the automatic speech recognition module. The acoustic feature data includes, but is not limited to, one or more of pitch, energy, and speech rate and their statistical features.

[0032] Specifically, by having two modules, automatic speech recognition and audio processing, work in parallel, streaming text data and acoustic features such as pitch, energy, and speech rate, along with their statistics, are extracted from the user's original speech in real time. This constructs the data foundation for multimodal analysis and provides comprehensive input for subsequent fusion processing.

[0033] In some embodiments, step S20 specifically includes the following steps: mapping each round of response text generated by the dialogue module to predefined behavior tags using preset mapping rules, and generating a behavior tag sequence in chronological order. The mapping rules are implemented based on keyword matching, intent recognition models, or regular expressions. Within the real-time sliding dialogue window, the behavior tag sequence is matched with predefined abnormal interaction patterns in the abnormal pattern rule base. Abnormal interaction patterns include at least rigid repetition patterns and invalid loop patterns. The rigid repetition pattern is defined as: within a preset number of consecutive dialogue rounds, the frequency of occurrence of the same behavior tag exceeds a first threshold. The invalid loop pattern is defined as: within a preset number of consecutive dialogue rounds, the number of times a specific behavior tag subsequence is repeated exceeds a second threshold.

[0034] Furthermore, the rules and their thresholds in the abnormal pattern rule base are dynamically configured through statistical analysis of historical abnormal dialogue data. The statistical analysis includes: calculating the correlation between the frequency of occurrence of specific behavior tags or behavior sequences in historical dialogues and the final dialogue failure rate; and adaptively adjusting the first threshold for triggering rigid repetitive patterns or the second threshold for triggering invalid loop patterns based on the correlation analysis results.

[0035] Specifically, the system abstracts AI responses into behavioral label sequences in real time through preset mapping rules, and matches them with an anomaly pattern rule base using a sliding dialogue window. The core patterns include "rigid repetition pattern" which determines that a specific behavioral label appears too frequently in consecutive rounds, and "invalid loop pattern" which determines that the number of repetitions of a specific behavioral subsequence exceeds the limit. Furthermore, the system introduces a data-driven adaptive optimization mechanism, which dynamically adjusts the frequency threshold in the rule base by statistically analyzing the correlation between the frequency of specific behaviors or sequences in historical dialogues and the final dialogue failure rate. This allows the anomaly detection standards to become increasingly accurate as business data accumulates, effectively improving the real-time identification of process-related and structured dialogue anomalies.

[0036] In some embodiments, step S30 specifically includes the following steps: inputting time-aligned user speech text and acoustic feature data into a pre-trained multimodal emotion classification model, performing joint inference by the multimodal emotion classification model to obtain a negative emotion score that represents the user's current negative emotion level with continuous values; using a pre-trained language model to generate text embedding vectors of the user's previous speech and the current response of the dialogue module, and obtaining a semantic relevance score by calculating the cosine similarity between the two text embedding vectors.

[0037] Furthermore, the multimodal emotion classification model is trained: a training dataset is obtained, which contains a large number of dialogue samples labeled with negative emotion tags. Each sample contains user speech text, time-aligned acoustic feature data, and the corresponding real emotion tag; a multimodal fusion network is constructed based on deep learning. The multimodal fusion network includes at least a text encoder for processing text features, an acoustic encoder for processing acoustic features, and a fusion classifier for fusing bimodal information; supervised learning is then used to train the multimodal fusion network end-to-end with the goal of minimizing the difference between the model's prediction results and the real emotion tags until the model converges.

[0038] Specifically, a pre-trained multimodal emotion classification model is used to jointly infer time-aligned user text and acoustic features to generate continuously valued negative emotion scores, effectively capturing the comprehensive emotional information contained in the text content and speech tone. At the same time, a pre-trained language model is used to generate text embedding vectors of the user's previous utterance and the AI's current response, respectively. The semantic relevance score is obtained by calculating the cosine similarity between the two, accurately assessing the contextual fit of the response. This multimodal emotion classification model is trained on a dialogue sample dataset containing labeled emotion tags. It is trained end-to-end through a deep learning network containing a text encoder, an acoustic encoder, and a fusion classifier, with the goal of minimizing prediction error until convergence, thereby ensuring that the model can accurately understand and quantify the user's true emotional state during the interaction process.

[0039] In some embodiments, step S40 specifically includes the following steps: when the anomaly detection result indicates that at least one of the following situations has occurred: an anomaly or negative emotion score exceeds a preset score threshold or semantic relevance score is lower than a preset similarity threshold, it is determined that the anomaly triggering condition is met, and an anomaly marker is generated. The anomaly marker is a structured data object, which includes at least the anomaly triggering time, anomaly level, the type of anomaly triggered, and the associated dialogue context text that triggered the anomaly.

[0040] Specifically, by integrating multi-dimensional signals, when any of the following conditions are detected—behavioral abnormality, emotion score exceeding the threshold, or semantic relevance falling below the threshold—it is determined to be an interaction abnormality, and a structured tag containing timestamp, abnormality level, type, and associated dialogue context is generated.

[0041] The implementation principle of a real-time dialogue anomaly detection method based on multi-dimensional feature fusion in this application is as follows: During the dialogue, the system receives in parallel a multimodal data stream consisting of streaming text generated by ASR and acoustic features extracted from user speech. Subsequently, the data stream is synchronously sent to two analysis channels: one is behavioral pattern analysis, which abstracts AI responses into predefined behavioral labels and forms sequences, and uses an anomaly pattern rule base for real-time matching to identify structural anomalies such as rigid repetition and invalid loops; the other is emotion and semantic joint analysis, which calculates negative emotion scores by fusing text and acoustic features through a multimodal emotion classification model, and obtains semantic relevance scores by calculating the cosine similarity of the embedding vectors of the two parties' statements; finally, the above behavioral anomaly signals, emotion scores and semantic relevance scores are fused, and a comprehensive decision is made according to preset logic. Once the conditions are met, a structured anomaly label is generated. By using multimodal parallel processing and feature fusion, real-time detection of interaction anomalies is achieved, overcoming the lag of traditional offline quality inspection. At the same time, by introducing the analysis of user voice acoustic features, the shortcomings of pure text analysis are made up for, and the true emotions conveyed by users through tone of voice and other means can be captured, making emotion judgment more accurate, thereby significantly improving the automation level and depth of dialogue monitoring.

[0042] Figure 1 This is a flowchart illustrating a real-time dialogue anomaly detection method based on multi-dimensional feature fusion in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows; unless explicitly stated herein, there is no strict order requirement for the execution of these steps, and they can be executed in other orders; and Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but may be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0043] Based on the same technical concept, referring to Figure 2 This application also provides a real-time dialogue anomaly detection device based on multi-dimensional feature fusion, which adopts the following technical solution: The device includes: The speech recognition module is used to receive and recognize streaming text data generated from the user's original speech, and to extract acoustic feature data from the user's original speech in real time. The behavior determination module is used to perform behavior abstraction to form a behavior label sequence based on the responses generated by the dialogue module in the streaming text data, and to perform real-time matching of the behavior label sequence based on the preset abnormal pattern rule base to obtain the abnormal detection results of the behavior pattern. The semantic analysis module is used to calculate and output negative emotion scores based on user speech text and time-aligned acoustic feature data in streaming text data, through a preset multimodal emotion classification model; at the same time, it calculates the similarity between the text embedding vector of the current response of the dialogue module and the user's previous speech to obtain a semantic relevance score. The anomaly labeling module is used to integrate anomaly detection results, negative sentiment scores, and semantic relevance scores. It performs a comprehensive evaluation based on a preset fusion judgment logic, and generates an anomaly label when the anomaly triggering conditions are met.

[0044] In some embodiments, the speech recognition module is specifically used to perform real-time recognition of the user's original speech using an automatic speech recognition module, generating streaming text data organized by dialogue turns; and... Acoustic feature data is extracted from the user's original speech in real time through an independent audio processing module and an automatic speech recognition module. The acoustic feature data includes, but is not limited to, one or more of pitch, energy, and speech rate and their statistical features.

[0045] In some embodiments, the behavior determination module is specifically used to map each round of response text generated by the dialogue module to a predefined behavior label through a preset mapping rule, and generate a behavior label sequence in chronological order. The mapping rule is implemented based on keyword matching, intent recognition model or regular expression. Within a real-time sliding dialog window, the sequence of behavior tags is matched with predefined abnormal interaction patterns in the abnormal pattern rule base. Abnormal interaction patterns include at least rigid repetition patterns and invalid loop patterns. The rigid repetitive mode is defined as: within a preset number of consecutive dialogue rounds, the frequency of the same behavior label exceeds a first threshold; the invalid loop mode is defined as: within a preset number of consecutive dialogue rounds, the number of times a specific behavior label subsequence is repeated exceeds a second threshold.

[0046] In some embodiments, the behavior determination module is further configured to dynamically configure the rules and their thresholds in the abnormal pattern rule base by performing statistical analysis on historical abnormal dialogue data. The statistical analysis includes: calculating the correlation between the frequency of occurrence of a specific behavior label or behavior sequence in historical dialogue and the final dialogue failure rate; and adaptively adjusting the first threshold for triggering a rigid repetitive pattern or the second threshold for triggering an invalid loop pattern based on the correlation analysis results.

[0047] In some embodiments, the semantic analysis module is specifically used to input time-aligned user speech text and acoustic feature data into a pre-trained multimodal emotion classification model, and the multimodal emotion classification model performs joint inference to obtain a continuously valued negative emotion score that represents the user's current negative emotion level. The pre-trained language model is used to generate text embedding vectors for the user's previous utterance and the current response from the dialogue module, and the semantic relevance score is obtained by calculating the cosine similarity between the two text embedding vectors.

[0048] In some embodiments, the semantic analysis module is also used to train a multimodal emotion classification model: to obtain a training dataset containing a large number of dialogue samples labeled with negative emotion tags, each sample containing user speech text, time-aligned acoustic feature data and corresponding real emotion tags. A multimodal fusion network is constructed based on deep learning. The multimodal fusion network includes at least a text encoder for processing text features, an acoustic encoder for processing acoustic features, and a fusion classifier for fusing bimodal information. Supervised learning is used to train the multimodal fusion network end-to-end with the goal of minimizing the difference between the model's predictions and the actual sentiment labels, until the model converges.

[0049] In some embodiments, the anomaly labeling module is specifically used to determine that the anomaly triggering condition is met and generate an anomaly label when the anomaly detection result indicates that at least one of the following occurs: an anomaly or a negative emotion score exceeding a preset score threshold or a semantic relevance score falling below a preset similarity threshold. The anomaly label is a structured data object that includes at least the anomaly triggering time, anomaly level, the type of anomaly triggered, and the associated dialogue context text that triggered the anomaly.

[0050] This application also discloses a control device.

[0051] Specifically, the control device includes a memory and a processor, with the memory storing a computer program that can be loaded by the processor and executed as described above for real-time detection of dialogue anomalies based on multi-dimensional feature fusion.

[0052] This application also discloses a computer-readable storage medium.

[0053] Specifically, the computer-readable storage medium stores a computer program that can be loaded by a processor and executed, such as the real-time dialogue anomaly detection method based on multi-dimensional feature fusion described above. The computer-readable storage medium includes, for example, various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0054] The above are all preferred embodiments of this application and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A real-time dialogue anomaly detection method based on multi-dimensional feature fusion, characterized in that, include: Receive and recognize streaming text data generated from the user's original speech, and extract acoustic feature data from the user's original speech in real time; Based on the responses generated by the dialogue module in the streaming text data, behavior abstraction is performed to form a behavior tag sequence, and the behavior tag sequence is matched in real time based on a preset abnormal pattern rule base to obtain the abnormal detection results of the behavior pattern. Based on the user speech text in the streaming text data and the time-aligned acoustic feature data, a negative emotion score is calculated and output through a preset multimodal emotion classification model; at the same time, the similarity between the text embedding vector of the current response of the dialogue module and the user's previous speech is calculated to obtain a semantic relevance score. The anomaly detection results, the negative emotion score, and the semantic relevance score are integrated and comprehensively evaluated according to a preset fusion judgment logic. When the anomaly triggering condition is met, an anomaly marker is generated.

2. The real-time dialogue anomaly detection method based on multi-dimensional feature fusion according to claim 1, characterized in that, The process of receiving and recognizing streaming text data generated from the user's original speech, and extracting acoustic feature data from the user's original speech in real time, includes: The system uses an automatic speech recognition module to recognize the user's original speech in real time, generating streaming text data organized according to dialogue turns; and... The acoustic feature data is extracted from the user's original speech in real time through an independent audio processing module and an automatic speech recognition module in parallel. The acoustic feature data includes, but is not limited to, one or more of pitch, energy, and speech rate and their statistical features.

3. The real-time dialogue anomaly detection method based on multi-dimensional feature fusion according to claim 1, characterized in that, The process involves abstracting behaviors based on responses generated by the dialogue module within the streaming text data to form a sequence of behavior tags, and then performing real-time matching of these behavior tag sequences based on a preset abnormal pattern rule base to obtain abnormal detection results for the behavior patterns. This includes: By using preset mapping rules, each round of response text generated by the dialogue module is mapped to a predefined behavior label, and the behavior label sequence is generated by arranging them in chronological order. The mapping rules are implemented based on keyword matching, intent recognition models, or regular expressions. Within a real-time sliding dialog window, the sequence of behavioral tags is matched with predefined abnormal interaction patterns in the abnormal pattern rule base, wherein the abnormal interaction patterns include at least rigid repetition pattern and invalid loop pattern. The rigid repetition mode is defined as: within a preset number of consecutive dialogue rounds, the frequency of occurrence of the same behavior tag exceeds a first threshold; the invalid loop mode is defined as: within a preset number of consecutive dialogue rounds, the number of times a specific behavior tag subsequence is repeated exceeds a second threshold.

4. The real-time dialogue anomaly detection method based on multi-dimensional feature fusion according to claim 3, characterized in that, Before generating an anomaly marker by fusing the anomaly detection result, the negative sentiment score, and the semantic relevance score, and performing a comprehensive evaluation based on a preset fusion judgment logic, the process further includes: The rules and thresholds in the abnormal pattern rule base are dynamically configured through statistical analysis of historical abnormal dialogue data. The statistical analysis includes: calculating the correlation between the frequency of occurrence of specific behavior tags or behavior sequences in historical dialogues and the final dialogue failure rate; and adaptively adjusting the first threshold for triggering the rigid repetition pattern or the second threshold for triggering the invalid loop pattern based on the correlation analysis results.

5. The real-time dialogue anomaly detection method based on multi-dimensional feature fusion according to claim 1, characterized in that, The acoustic feature data, which is based on user speech text and time alignment in the streaming text data, is used to calculate and output a negative emotion score through a preset multimodal emotion classification model. Simultaneously, the similarity between the text embedding vectors of the current response from the dialogue module and the user's previous utterance is calculated to obtain a semantic relevance score, including: The time-aligned user speech text and the acoustic feature data are jointly input into a pre-trained multimodal emotion classification model. The multimodal emotion classification model performs joint inference to obtain the negative emotion score, which is a continuously valued representation of the user's current negative emotion level. The pre-trained language model is used to generate text embedding vectors for the user's previous utterance and the current response from the dialogue module, and the semantic relevance score is obtained by calculating the cosine similarity between the two text embedding vectors.

6. The real-time dialogue anomaly detection method based on multi-dimensional feature fusion according to claim 5, characterized in that, Before inputting the time-aligned user speech text and the acoustic feature data into a pre-trained multimodal emotion classification model, the method further includes: Training the multimodal emotion classification model: Obtain a training dataset containing a large number of dialogue samples labeled with negative emotion tags. Each sample contains user speech text, time-aligned acoustic feature data, and corresponding real emotion tags. A multimodal fusion network is constructed based on deep learning. The multimodal fusion network includes at least a text encoder for processing text features, an acoustic encoder for processing acoustic features, and a fusion classifier for fusing bimodal information. Supervised learning is used to train the multimodal fusion network end-to-end with the goal of minimizing the difference between the model's predicted results and the actual sentiment labels, until the model converges.

7. The real-time dialogue anomaly detection method based on multi-dimensional feature fusion according to claim 5, characterized in that, The system integrates the anomaly detection results, the negative sentiment score, and the semantic relevance score, and performs a comprehensive evaluation based on a preset fusion judgment logic. When an anomaly triggering condition is met, an anomaly marker is generated, including: When the anomaly detection result indicates the presence of an anomaly, or when the negative emotion score exceeds a preset score threshold, or when the semantic relevance score is lower than a preset similarity threshold, the anomaly triggering condition is determined to be met, and the anomaly marker is generated. The anomaly marker is a structured data object, which includes at least the anomaly triggering time, anomaly level, the type of anomaly triggered, and the associated dialogue context text that triggered the anomaly.

8. A real-time dialogue anomaly detection device based on multi-dimensional feature fusion, characterized in that, The device includes: The speech recognition module is used to receive and recognize streaming text data generated from the user's original speech, and to extract acoustic feature data from the user's original speech in real time. The behavior determination module is used to perform behavior abstraction to form a behavior tag sequence based on the responses generated by the dialogue module in the streaming text data, and to perform real-time matching of the behavior tag sequence based on a preset abnormal pattern rule base to obtain the abnormal detection results of the behavior pattern. The semantic analysis module is used to calculate and output a negative emotion score based on the user's speech text in the streaming text data and the time-aligned acoustic feature data, through a preset multimodal emotion classification model; at the same time, it calculates the similarity between the text embedding vector of the current response of the dialogue module and the user's previous speech to obtain a semantic relevance score. The anomaly labeling module is used to fuse the anomaly detection results, the negative emotion score, and the semantic relevance score, and to perform a comprehensive evaluation based on a preset fusion judgment logic. When the anomaly triggering condition is met, an anomaly label is generated.

9. A control device, characterized in that, The device includes: A memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer program is stored that can be loaded by a processor and executed as described in any one of claims 1 to 7.

Images