Real-time interaction violation detection method, system and device and medium

Abstract

The invention relates to a real-time interaction violation detection method, system and device and a medium. The method comprises the following steps: acquiring an online interaction session original information flow containing a user text sequence, a voice signal, an image file and an interaction behavior timestamp; extracting text semantic vectors, voice acoustic features and image visual content description, and integrating to generate an initial feature vector set; based on the interaction behavior timestamps, constructing an interaction time sequence diagram by taking the initial vectors as nodes, calculating multi-modal association weights among the nodes and updating connection edges to obtain a multi-modal fusion diagram; and inputting the fused graph into a graph neural network, outputting a global graph embedded vector through message passing and node aggregation, matching a preset violation mode vector library to calculate a similarity score, determining a violation type, and generating a risk assessment conclusion containing the violation type and confidence. According to the method, cross-statement and cross-modal context violation association is effectively captured, violation judgment accuracy is improved, and an intervention basis is provided for a platform.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, and in particular relates to a real-time interactive violation detection method, system, device and medium. Background Technology

[0002] With the rapid development of internet communication technology, real-time interactive scenarios such as call centers, online customer service platforms, and live streaming have been widely applied in finance, e-commerce, and government affairs. Their interactive data exhibits significant characteristics of multimodality, high concurrency, and strong real-time requirements, making the need for compliance detection of interactive content increasingly urgent. Current violation detection methods for such scenarios have obvious limitations: on the one hand, traditional solutions often rely on single-modal keyword matching or simple rule-based judgment, failing to effectively integrate multimodal information such as speech semantics and image vision, and struggling to capture cross-sentence contextual violations (such as combinations of violating keywords appearing scattered throughout a dialogue), resulting in narrow detection coverage and a high rate of false positives and false negatives. On the other hand, existing systems mostly use batch processing for real-time streaming data, resulting in high response latency, making it difficult to meet the millisecond-level detection requirements of high-concurrency scenarios. Furthermore, updating violation detection rules requires a system restart, preventing dynamic effectiveness. Additionally, the lack of effective information isolation mechanisms when multiple call centers or multiple user interaction data are simultaneously accessed easily leads to data confusion, further reducing the accuracy of violation determination. Summary of the Invention

[0003] Therefore, it is necessary to provide a real-time interactive violation detection method, system, device, and medium that can achieve comprehensive coverage of multimodal data in real-time interactive scenarios, effectively capture contextual violation associations across sentences and modalities, and improve the accuracy of violation type determination, in order to address the above-mentioned technical problems.

[0004] Firstly, this application provides a real-time interactive violation detection method, including:

[0005] Obtain the complete raw information stream of the online interactive session; the raw information stream includes user text sequences, voice signals, image files, and timestamps of interactive behaviors.

[0006] Semantic vectors, acoustic features, and visual content descriptions are extracted from the text sequence, speech signal, and image file in the original information stream to generate an initial feature vector set.

[0007] Based on the interaction behavior timestamps, the vectors in the initial feature vector set are used as nodes to construct an interaction time sequence graph. Based on the interaction time sequence graph, the multimodal association weights are calculated and the node connection edges are updated to obtain a multimodal fusion graph.

[0008] The multimodal fusion graph is input into the graph neural network model, which outputs a global graph embedding vector. The similarity score between the global graph embedding vector and the preset violation pattern vector library is calculated to determine the violation type of the current interaction and generate an interaction risk assessment conclusion.

[0009] In one embodiment, semantic vectors, acoustic features, and visual content descriptions are extracted from the text sequence, speech signal, and image file in the original information stream, respectively, to generate an initial feature vector set, including:

[0010] The speech signal in the original information stream is processed in real time to convert the speech signal into text data, thus obtaining speech-transcribed text.

[0011] For the text sequence and speech-to-text in the original information stream, text semantic information is extracted using text feature extraction methods to generate corresponding text semantic vectors.

[0012] Acoustic attributes, including pitch, speech rate, and audio spectrum, are extracted from the untranscribed speech signal in the original information stream to obtain speech acoustic feature vectors.

[0013] Image recognition technology is used to analyze the visual elements in the image files in the original information stream, generate visual information describing the image content, and obtain the image visual content vector.

[0014] The text semantic vectors, speech acoustic feature vectors, and image visual content vectors are aggregated and integrated to form an initial feature vector set covering multimodal information.

[0015] In one embodiment, an interaction time sequence graph is constructed using vectors from an initial feature vector set as nodes based on the interaction behavior timestamps. Multimodal association weights are calculated based on the interaction time sequence graph, and node connection edges are updated to obtain a multimodal fusion graph, including:

[0016] Based on the interactive behavior timestamp, each feature vector in the initial feature vector set is bound to the corresponding time information to obtain a multimodal feature vector group with time stamp.

[0017] Using multimodal feature vector groups as nodes, a directed acyclic interaction time sequence diagram is constructed based on the temporal order of each node and the response relationship in real-time interaction. The interaction time sequence diagram structure is then divided into node clusters for different traffic flows.

[0018] For the constructed interaction time sequence graph, focus on adjacent nodes in the graph, calculate the degree of feature association between adjacent nodes through feature similarity, and obtain the multimodal association weights corresponding to the connecting edges of each node.

[0019] The multimodal association weights are assigned to the corresponding node connection edges in the interaction sequence graph, replacing the initial default connection strength. Based on the context requirements of real-time interaction, the association weights of nodes that are cross-statements, non-adjacent but logically related in the same session are supplemented and new node connection edges are added to obtain the edge structure of the interaction sequence graph.

[0020] By integrating and updating the edge structure and node cluster, a multimodal fusion graph is finally formed that combines time sequence, multimodal association characteristics, and call isolation functions.

[0021] In one embodiment, the multimodal association weights corresponding to the connection edges of each node are calculated using the following formula:

[0022]

[0023]

[0024] in, Represents a node and nodes Multimodal association weights between them Represents a node ,node Text semantic vectors, Represents a node ,node Speech acoustic vectors, Represents a node ,node Image visual vectors, , , Represents the modal weighting coefficient. Indicates feature similarity calculation, This represents the absolute value of the time difference between two nodes. This represents the time decay coefficient.

[0025] In one embodiment, a global graph embedding vector is output from the multimodal fusion graph input graph neural network model. The similarity score between the global graph embedding vector and a preset violation pattern vector library is calculated to determine the violation type of the current interaction, generating an interaction risk assessment conclusion, including:

[0026] The multimodal fusion graph is input into the graph neural network model, and the output is a global graph embedding vector representing the deep structure of the current interaction process; the deep structure includes time correlation, multimodal correlation and call isolation characteristics.

[0027] Calculate the cosine similarity between the global graph embedding vector and each violation pattern vector in the pre-established violation pattern vector library, and record the similarity scores; the violation pattern vector library includes preset keyword detection rules.

[0028] If any similarity score is higher than the preset threshold, the violation pattern category corresponding to the score is directly matched to determine the violation type of the current interaction.

[0029] For interactions whose violation types have been preliminarily determined, a confidence score corresponding to the violation type is calculated and generated by combining the priority and similarity score of the corresponding violation pattern.

[0030] By integrating violation types and confidence scores, and supplementing the corresponding call identification tags for the interactions, an interaction risk assessment conclusion is generated that includes violation type tags, confidence scores, and call identification tags.

[0031] In one embodiment, the confidence score is calculated using the following formula:

[0032]

[0033]

[0034] in, This represents the confidence score. This indicates the normalized priority of the violation pattern, and classifies rules according to the severity of the violation. This represents the combined similarity score between the global graph embedding vector and the preset violation pattern vector. , , These represent the weight coefficients of text modality, speech modality, and view modality in the overall similarity score, respectively. This represents the text modality score. The Drools rule engine, which uses the DRL language to construct complex matching conditions, performs violation feature matching on text sequences and speech-to-text transcripts, and then outputs the result. The speech modality score is calculated by comparing the cosine similarity between the current interaction's speech acoustic feature vector and a preset speech acoustic standard vector. The view modality score is calculated by comparing the cosine similarity between the visual content vector of the currently interacting image and the preset visual standard vector. Indicates priority weight. This represents the similarity weight.

[0035] Secondly, this application also provides a real-time interactive violation detection system, the system comprising:

[0036] The information flow acquisition module is used to acquire all the original information flow of the online interactive session; the original information flow includes user text sequences, voice signals, image files, and timestamps of interactive behaviors.

[0037] The feature extraction module is used to extract semantic vectors, acoustic features, and visual content descriptions from text sequences, speech signals, and image files in the original information stream, and generate an initial feature vector set.

[0038] The temporal sequence graph construction module is used to construct an interaction temporal sequence graph by using vectors in the initial feature vector set as nodes based on the interaction behavior timestamps. Based on the interaction temporal sequence graph, the multimodal association weights are calculated and the node connection edges are updated to obtain a multimodal fusion graph.

[0039] The violation assessment module is used to input the multimodal fusion graph into the graph neural network model and output the global graph embedding vector. It calculates the similarity score between the global graph embedding vector and the preset violation pattern vector library, determines the violation type of the current interaction, and generates an interaction risk assessment conclusion.

[0040] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0041] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned method.

[0042] The aforementioned real-time interactive violation detection method, system, computer equipment, and storage medium acquire all the original information streams of an online interactive session. These original information streams include user text sequences, voice signals, image files, and interaction behavior timestamps. Semantic vectors are extracted from the text sequences, acoustic features from the voice signals, and visual content descriptions from the image files. These semantic vectors, acoustic features, and visual content descriptions are integrated to generate an initial feature vector set. Based on the interaction behavior timestamps, each vector in the initial feature vector set is used as a node. An interaction time sequence graph is constructed according to the temporal order of each node during the interaction process. Multimodal association weights between nodes are calculated based on this interaction time sequence graph, and the connection edges between nodes are updated using these weights to obtain a multimodal fusion graph. The multimodal fusion graph is input into a graph neural network model. Through message passing and node aggregation operations, the model outputs a global graph embedding vector representing the deep structure of the current interaction process. The similarity score between the global graph embedding vector and each violation pattern vector in a preset violation pattern vector library is calculated. The violation type of the current interaction is determined based on whether the score is higher than a preset threshold. Finally, an interaction risk assessment conclusion containing the violation type and confidence level is generated. This method achieves comprehensive coverage of multimodal data in real-time interactive scenarios through coherent processing of multimodal feature extraction, temporal graph construction and fusion, graph embedding calculation, and similarity matching. It effectively captures contextual violation associations across sentences and modalities. Simultaneously, it relies on interaction behavior timestamps to ensure data isolation between different interactive sessions, avoiding data confusion. By learning the deep structure of interactions through a graph neural network model and matching it with a violation pattern vector library, the accuracy of violation type determination is improved. The generated interaction risk assessment conclusions can directly provide platform administrators with clear intervention criteria, adapting to the compliance detection needs of various real-time interactive scenarios such as call centers, online customer service, and live streaming rooms. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 A flowchart of a real-time interactive violation detection method provided in an embodiment of the present invention;

[0045] Figure 2 This is a structural block diagram of a real-time interactive violation detection system provided in an embodiment of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0047] In one embodiment, such as Figure 1 As shown, this application provides a real-time interactive violation detection method, which may include the following steps:

[0048] Step S101: Obtain the complete raw information stream of the online interactive session; the raw information stream includes user text sequences, voice signals, image files, and timestamps of interactive behaviors.

[0049] Specifically, for real-time interactive scenarios such as call centers, online customer service platforms, and live streaming, the system connects to the data stream of the interactive system through a data interface to collect all the raw information streams generated in each interactive session. This includes user text sequences such as user-inputted text messages and system-automated replies; audio signals such as user voice input and customer service voice responses; image files such as user-sent images and screenshots from live streams; and interaction timestamps that precisely record the generation or transmission time of each piece of text, each piece of audio, and each image file. These timestamps will serve as a crucial basis for "time-sequence correlation" and "information isolation" in subsequent processing.

[0050] Step S102: Extract semantic vectors, acoustic features, and visual content descriptions from the text sequence, speech signal, and image file in the original information stream to generate an initial feature vector set.

[0051] Feature extraction is performed on multimodal data in the original information stream, transforming unstructured / semi-structured data into computable vector forms. The specific processing steps are as follows: For text sequences, pre-trained language models (such as BERT and RoBERTa) are used to semantically encode the text content, extracting semantic vectors that represent the core meaning of the text, ensuring that keywords and semantic logic are effectively captured. For speech signals, acoustic features such as pitch, speech rate, and audio spectrum are extracted using audio processing algorithms (such as MFCC and Mel-frequency spectral analysis), quantizing these features into acoustic feature vectors to reflect the emotional tendency or specific speech keyword-related features. For image files, image recognition models (such as ResNet and ViT) are used to analyze visual elements such as objects, text, and scenes in the image, generating visual content descriptions of the image content, which are then transformed into visual content vectors through vector mapping. Finally, the extracted text semantic vectors, speech acoustic feature vectors, and image visual content vectors are summarized to form an initial feature vector set covering multimodal information. Each vector is associated with the interaction timestamp of the original data, ensuring that its temporal attributes can be traced subsequently.

[0052] Step S103: Based on the interaction behavior timestamp, the vectors in the initial feature vector set are used as nodes to construct an interaction time sequence graph. Based on the interaction time sequence graph, the multimodal association weights are calculated and the node connection edges are updated to obtain a multimodal fusion graph.

[0053] First, each vector in the initial feature vector set is used as a node in the graph. Combined with the timestamps of the acquired interaction behaviors, the node's position in the graph is determined according to the chronological order of the data corresponding to each node. Directed edges are constructed based on the response relationships in real-time interaction (such as the correspondence between user-sent text vectors and system-reply text vectors, and the accompanying relationship between voice vectors and concurrent image vectors) to form a preliminary interaction time sequence graph. At the same time, for scenarios with multiple callers or multiple users interacting in parallel, nodes are partitioned according to the interaction session identifiers (such as caller ID and user ID) to ensure that the node clusters of different sessions are independent of each other and to achieve "information isolation". Subsequently, based on the vector features of each node in the interaction sequence graph, the degree of association between adjacent nodes (including intramodal and cross-modal adjacent nodes) is quantified through feature similarity calculation (such as cosine similarity) to obtain multimodal association weights. These weights are then assigned to the connecting edges of the corresponding nodes, replacing the initial default edge weights, thereby reflecting the tightness of the association between nodes. For nodes that are logically related across sentences in the same session (such as the text vector of sentence N and the speech vector of sentence N+1 in the same session), additional association weights are calculated and connecting edges are added to improve the graph structure. Finally, this step yields a multimodal fusion graph that integrates the characteristics of "temporal order," "multimodal association," and "information isolation."

[0054] Step S104: Input the multimodal fusion graph into the graph neural network model and output the global graph embedding vector. Calculate the similarity score between the global graph embedding vector and the preset violation pattern vector library, determine the violation type of the current interaction, and generate an interaction risk assessment conclusion.

[0055] The multimodal fusion graph is input into a graph neural network model (such as GCN or GAT). The model passes the feature information of each node to its neighboring nodes through a message passing mechanism. Simultaneously, it integrates the feature information of all nodes in the graph through node aggregation operations (such as mean aggregation and attention aggregation), ultimately outputting a global graph embedding vector that represents the deep structure of the entire interaction process (including temporal association, multimodal association, and session isolation characteristics). This vector comprehensively reflects the overall characteristics of the interaction. Next, a pre-defined violation pattern vector library is invoked. This library stores standard vectors corresponding to various violation scenarios (such as standard vectors for "complaint + CBRC" violation scenarios, standard vectors for abusive violation scenarios, etc., with vector generation methods consistent with the global graph embedding vector). The cosine similarity algorithm is used to calculate the similarity score between the current interaction's global graph embedding vector and each violation pattern vector in the library, recording each score and its corresponding violation pattern category. Subsequently, a high-risk intent is determined based on a pre-defined threshold (such as 0.8): if a similarity score is higher than the threshold, the violation pattern category corresponding to that score is directly matched to determine the violation type of the current interaction; if all scores are lower than the threshold, it is determined as "no clear violation type". Finally, by combining the specific numerical value of the similarity score (the higher the score, the higher the confidence level) with the priority of the violation pattern, the confidence score corresponding to the violation type is calculated, and the violation type and confidence score are integrated to generate an interactive risk assessment conclusion that includes "violation type - confidence level".

[0056] The aforementioned real-time interactive violation detection method acquires the original information stream of online interactive sessions containing user text sequences, voice signals, image files, and interaction behavior timestamps; extracts text semantic vectors, voice acoustic features, and image visual content descriptions, and integrates them to generate an initial feature vector set; based on the interaction behavior timestamps, constructs an interaction time sequence graph with the initial vectors as nodes, calculates the multimodal association weights between nodes and updates the connection edges, obtaining a multimodal fusion graph; inputs the fusion graph into a graph neural network, outputs a global graph embedding vector through message passing and node aggregation, matches it with a preset violation pattern vector library to calculate a similarity score, determines the violation type, and generates a risk assessment conclusion containing the violation type and confidence level. This method achieves comprehensive coverage of multimodal data through multimodal feature extraction, time sequence graph processing, and graph embedding matching, effectively capturing cross-statement / cross-modal contextual violation associations; relies on timestamps to ensure the isolation of interactive data, and combines graph neural networks to improve the accuracy of violation judgment. The assessment conclusions can provide a basis for platform intervention and are suitable for real-time interactive compliance detection scenarios such as call centers, online customer service, and live streaming rooms.

[0057] In one embodiment, extracting semantic vectors, acoustic features, and visual content descriptions from the text sequence, speech signal, and image file in the original information stream to generate an initial feature vector set may include the following steps:

[0058] Step S201: Perform real-time transcription processing on the speech signal in the original information stream, convert the speech signal into text data, and obtain speech-transcribed text.

[0059] Step S202: Extract semantic information from the text sequence and speech-to-text in the original information stream using text feature extraction methods to generate corresponding text semantic vectors.

[0060] Step S203: Extract acoustic attributes, including pitch, speech rate, and audio spectrum, from the untranscribed speech signal in the original information stream to obtain the speech acoustic feature vector.

[0061] Step S204: Analyze the visual elements in the image file in the original information stream using image recognition technology to generate visual information describing the image content, and obtain the image visual content vector.

[0062] Step S205: Summarize the text semantic vector, speech acoustic feature vector, and image visual content vector, and integrate them to form an initial feature vector set covering multimodal information.

[0063] First, ASR technology is used to perform real-time transcription of the speech signal in the original information stream, converting the speech signal into text data that can participate in text analysis, thus obtaining speech-transcribed text. Second, for the text sequence directly input by the user in the original information stream and the speech-transcribed text obtained in the above steps, text feature extraction methods such as pre-trained language model encoding are used to extract the core semantic information of the two types of text, respectively, quantizing the semantic information into structured data and generating corresponding text semantic vectors. Third, for the untranscribed original speech signal in the original information stream, audio signal processing algorithms are used to extract its pitch, speech rate, audio spectrum, and other acoustic features. The acoustic attributes are converted into vector form to obtain speech acoustic feature vectors. Then, for image files in the original information stream, visual elements such as text, objects, and scenes in the images are analyzed using image recognition techniques such as visual model analysis to generate visual information describing the core content of the images. The visual information is then mapped into vectors to obtain image visual content vectors. Finally, the text semantic vectors, speech acoustic feature vectors, and image visual content vectors generated in the above steps are summarized, and the interaction behavior timestamps (from the original information stream) corresponding to each vector are associated with them to form an initial feature vector set covering multimodal information of text, speech, and images.

[0064] This embodiment's feature extraction process is adapted to the core requirements of violation detection in real-time interactive scenarios, possessing multi-dimensional technical value: First, by converting speech signals into text through ASR transcription, it ensures that keyword information in the speech can participate in subsequent detection in conjunction with the original text sequence, avoiding the omission of speech modal information; second, it achieves full coverage of multi-modal features, extracting both text semantic vectors to capture the logical association of keywords and speech acoustic features (which can assist in identifying emotionally charged violations) and image visual content vectors (which can detect images containing violation elements), breaking through the limitations of traditional single text detection; third, it associates the timestamps of interactive behaviors in the feature integration stage, providing time dimension support for subsequent construction of interaction sequence graphs, achieving isolation of multiple call information and contextual association detection within the same call; fourth, the generated initial feature vector set can be directly used as node material for subsequent graph structure construction, laying the foundation for learning deep interactive features through graph neural networks and improving the accuracy of violation judgment, adapting to the real-time detection and dynamic rule update requirements in high-concurrency scenarios.

[0065] In one embodiment, an interaction time sequence graph is constructed by using vectors from the initial feature vector set as nodes based on the interaction behavior timestamps. Multimodal association weights are calculated based on the interaction time sequence graph, and node connection edges are updated to obtain a multimodal fusion graph. This process may include the following steps:

[0066] Step S301: Based on the interaction behavior timestamp, bind each feature vector in the initial feature vector set with the corresponding time information to obtain a multimodal feature vector group with time stamp.

[0067] Step S302: Using the multimodal feature vector group as nodes, construct a directed acyclic interaction time sequence diagram based on the temporal order of each node and the response relationship in real-time interaction, and divide the node clusters of different traffic in the interaction time sequence diagram structure.

[0068] Step S303: Focus on adjacent nodes in the constructed interaction time sequence graph, calculate the feature association degree between adjacent nodes through feature similarity, and obtain the multimodal association weights corresponding to the connecting edges of each node.

[0069] Step S304: Assign the multimodal association weights to the corresponding node connection edges in the interaction sequence graph, replace the initial default connection strength, and, in combination with the context requirements of real-time interaction, supplement the association weights and add node connection edges for nodes that are cross-statements, non-adjacent but logically related in the same session, to obtain the edge structure of the interaction sequence graph.

[0070] Step S305: Integrate the updated edge structure and node cluster to finally form a multimodal fusion graph that combines time sequence, multimodal association characteristics and traffic isolation functions.

[0071] Specifically, based on the timestamps of interactive behaviors, the text semantic vectors, speech acoustic feature vectors, and image visual content vectors in the initial feature vector set are bound to the time information of the corresponding original data, clarifying the time positioning of each vector in the interaction process, resulting in a time-stamped multimodal feature vector group. Secondly, using each vector in this multimodal feature vector group as a node, and based on the temporal order of each node, combined with the response relationship between the user and the system in real-time interaction (such as the correspondence between user input vectors and system feedback vectors, and the accompanying relationship of concurrent multimodal data), a directed acyclic interaction time sequence graph is constructed. Simultaneously, different call node clusters are divided according to call ID or user ID to achieve node isolation for different interactive sessions. Thirdly, focusing on the constructed interaction time sequence graph, the adjacent nodes in the graph are... (Including contiguous nodes of the same modality and concurrent nodes across modalities), the feature correlation between each node is quantified using feature similarity calculation methods such as cosine similarity to obtain the multimodal correlation weights corresponding to the connection edges of each node. Subsequently, the calculated multimodal correlation weights are assigned to the corresponding node connection edges in the interaction time sequence graph, replacing the initial default unified connection strength. At the same time, in combination with the context detection requirements of real-time interaction, for nodes that are cross-statements, non-adjacent but logically related in the same session (such as the vectors corresponding to the illegal keywords that appear scattered in the same session), the correlation weights are supplemented and node connection edges are added to improve the edge structure of the time sequence graph. Finally, the updated node connection edge structure and the divided node clusters are integrated to form a multimodal fusion graph that integrates time sequence, multimodal correlation characteristics and call isolation functions.

[0072] This embodiment ensures the accuracy of interaction timing and achieves isolation of multiple call information through timestamp binding and node cluster partitioning, avoiding confusion of different session data and meeting the design requirements of "parallel processing of multiple calls" and "independent and non-interfering information". Secondly, through multimodal association weight calculation and edge structure optimization, it breaks through the limitation of traditional time-series processing that only focuses on time order, and can effectively capture cross-modal and cross-statement feature associations, providing support for subsequent accurate identification of scattered and related violations and solving the problem of "context keyword association detection". Thirdly, the final multimodal fusion graph integrates the three core dimensions of time, multimodal features, and call isolation, which can be directly used as input to the graph neural network, laying the foundation for subsequent extraction of global interaction features and improving the accuracy of violation judgment. Fourthly, each link of the process is adapted to real-time stream processing logic, and weight calculation and edge structure optimization can be executed efficiently, which can meet the performance requirements of millisecond-level detection in high-concurrency scenarios, while providing stable graph structure support for subsequent dynamic rule updates.

[0073] In one embodiment, the multimodal association weights corresponding to the connecting edges of each node can be calculated using the following formula:

[0074]

[0075]

[0076] in, Represents a node and nodes Multimodal association weights between them Represents a node ,node Text semantic vectors, Represents a node ,node Speech acoustic vectors, Represents a node ,node Image visual vectors, , , Represents the modal weighting coefficient. Indicates feature similarity calculation, This represents the absolute value of the time difference between two nodes. This represents the time decay coefficient.

[0077] This embodiment's multimodal association weight calculation formula integrates the similarity of text, speech, and image features and introduces a time decay mechanism. This not only balances the influence of different modal information through modal weight coefficients, comprehensively quantifies the degree of multimodal association between nodes, and avoids association omissions caused by single-modal analysis, but also uses the time decay factor to reflect the temporal characteristics of real-time interaction, weakening false associations of nodes with large time differences and improving the rationality of association weights. The calculation results can provide accurate basis for updating the strength of the connection edges of nodes in the interaction temporal graph, helping to build a multimodal fusion graph that is more in line with the actual interaction logic. This lays a reliable foundation for subsequent graph neural network extraction of global features and accurate determination of violation types, effectively adapting to the multimodal and temporal violation detection needs in real-time interaction scenarios.

[0078] In one embodiment, the multimodal fusion graph is input into the graph neural network model, which outputs a global graph embedding vector. The similarity score between the global graph embedding vector and a preset violation pattern vector library is calculated to determine the violation type of the current interaction and generate an interaction risk assessment conclusion. This may include the following steps:

[0079] Step S401: Input the multimodal fusion graph into the graph neural network model and output a global graph embedding vector representing the deep structure of the current interaction process; the deep structure includes time correlation, multimodal correlation and traffic isolation characteristics.

[0080] Step S402: Calculate the cosine similarity between the global graph embedding vector and each violation pattern vector in the pre-established violation pattern vector library, and record the similarity scores; the violation pattern vector library includes preset keyword detection rules.

[0081] Step S403: If any similarity score is higher than the preset threshold, the violation pattern category corresponding to the score is directly matched to determine the violation type of the current interaction.

[0082] Step S404: For interactions whose violation types have been preliminarily determined, calculate and generate the confidence score corresponding to the violation type by combining the priority and similarity score of the corresponding violation pattern.

[0083] Step S405: Integrate the violation type and confidence score, supplement the corresponding call identification for the interaction, and generate an interaction risk assessment conclusion that includes violation type label, confidence score and call identification.

[0084] Specifically, a multimodal fusion graph integrating time correlation, multimodal correlation, and traffic isolation characteristics is input into a graph neural network model. The model's message passing mechanism transmits node features, and node aggregation operations integrate the entire graph information, outputting a global graph embedding vector representing the deep structure of the current interaction process (including time correlation, multimodal correlation, and traffic isolation characteristics). Subsequently, a pre-established violation pattern vector library (containing standard vectors of various violation scenarios constructed based on preset keyword detection rules) is called. A cosine similarity algorithm is used to calculate the similarity between the global graph embedding vector and each violation pattern vector in the library, and all similarity scores are recorded. Each score is compared with a preset threshold. If... If any similarity score exceeds the threshold, the violation pattern category corresponding to that score is directly matched to preliminarily determine the violation type of the current interaction. For interactions with preliminarily determined violation types, the confidence score corresponding to the violation type is calculated by combining the preset priority of the violation pattern (e.g., the priority of patterns involving regulatory-related keywords is higher than that of ordinary violation patterns) and the corresponding similarity score through weight allocation. Finally, the preliminarily determined violation types and the calculated confidence scores are integrated, and the corresponding call identifiers for the interaction are added (to distinguish different parallel interaction sessions), generating a complete interaction risk assessment conclusion that includes violation type tags, confidence scores, and call identifiers.

[0085] This embodiment leverages graph neural networks to deeply mine features within the multimodal fusion graph, ensuring that the global graph embedding vector comprehensively captures the time sequence of interactions, multimodal associations, and session isolation attributes, providing accurate feature support for violation determination. Utilizing a violation pattern vector library containing keyword detection rules, it achieves rapid matching of violation types. By combining pattern priority and similarity scores to calculate confidence, it effectively balances the impact of violation severity and feature matching degree on the determination results, improving the reliability of violation determination. Supplementing with call traffic identifiers enables the evaluation conclusions to accurately relate to specific interactive sessions, ensuring information independence during parallel processing of multiple sessions while facilitating subsequent tracing and intervention of violating sessions. It adapts to the compliance detection needs of high-concurrency real-time interactive scenarios, providing management with accurate and implementable evidence for violation intervention.

[0086] In one embodiment, the confidence score can be calculated using the following formula:

[0087]

[0088]

[0089] in, This represents the confidence score. This indicates the normalized priority of the violation pattern, and classifies rules according to the severity of the violation. This represents the combined similarity score between the global graph embedding vector and the preset violation pattern vector. , , These represent the weight coefficients of text modality, speech modality, and view modality in the overall similarity score, respectively. This represents the text modality score. The Drools rule engine, which uses the DRL language to construct complex matching conditions, performs violation feature matching on text sequences and speech-to-text transcripts, and then outputs the result. The speech modality score is calculated by comparing the cosine similarity between the current interaction's speech acoustic feature vector and a preset speech acoustic standard vector. The view modality score is calculated by comparing the cosine similarity between the visual content vector of the currently interacting image and the preset visual standard vector. Indicates priority weight. This represents the similarity weight.

[0090] The confidence score calculation formula in this embodiment achieves both accurate quantification of multimodal violation features and ensures the reliability and scenario adaptability of the judgment results, by introducing normalization priority. Cosine similarity score The weighted fusion logic, combining priority weights and similarity weights, allows for flexible adjustment of the influence ratio of both factors, enabling both... Quantifying the severity of different violation patterns highlights the weight of high-priority violations (such as those involving regulatory violations) in the determination, and can also leverage... This reflects the accuracy of feature matching between the global graph embedding vector and the violation pattern vector, avoiding result bias caused by single-dimensional judgment; text modality scoring is introduced. Speech modality score View Modal Score The system independently calculates violation features for three modalities. The text modality relies on the Drools Rule Language (DRL) to construct a Drools rule engine with complex matching conditions (including multi-keyword logical combinations, contextual constraints, and semantic conflict checks). This overcomes the limitations of traditional single-keyword matching and can accurately capture complex violation features such as multi-keyword logical combinations and contextual relationships in text sequences and speech-to-text transcriptions. The speech and view modalities use cosine similarity calculations to maintain the rigor of feature vector matching, ensuring accurate quantification of acoustic and visual violation features. The calculation results are output as percentages, intuitively presenting the reliability of violation judgments. This provides a quantitative basis for the final confirmation of violation types, further improving the accuracy and reference value of interaction risk assessment conclusions, and meeting the core requirements for accuracy and interpretability of violation judgments in real-time interaction scenarios.

[0091] In one embodiment, such as Figure 2 As shown, this application also provides a real-time interactive violation detection system, which may include:

[0092] The information flow acquisition module 501 is used to acquire all the original information flow of the online interactive session; the original information flow includes user text sequences, voice signals, image files and timestamps of interactive behaviors.

[0093] The feature extraction module 502 is used to extract semantic vectors, acoustic features and visual content descriptions from the text sequence, speech signal and image file in the original information stream, respectively, and generate an initial feature vector set.

[0094] The temporal sequence graph construction module 503 is used to construct an interaction temporal sequence graph by using vectors in the initial feature vector set as nodes based on the interaction behavior timestamps, calculate multimodal association weights based on the interaction temporal sequence graph and update the node connection edges to obtain a multimodal fusion graph.

[0095] The violation assessment module 504 is used to input the multimodal fusion graph into the graph neural network model and output the global graph embedding vector, calculate the similarity score between the global graph embedding vector and the preset violation pattern vector library, determine the violation type of the current interaction, and generate an interaction risk assessment conclusion.

[0096] The aforementioned real-time interactive violation detection system includes an information flow acquisition module that interfaces with real-time interactive scenarios such as call centers and online customer service. This module collects all the original information flow from online interactive sessions. The original information flow includes user text sequences, voice signals, image files, and timestamps recording the generation or transmission time of each data point, providing a complete data source for subsequent processing. A multimodal feature extraction module receives the original information flow output by the information flow acquisition module and extracts semantic vectors from the text sequences, acoustic features from the voice signals, and visual content descriptions from the image files. These three types of features are quantized into vector form and then aggregated to generate an initial feature vector set covering multimodal information. A time-series graph construction module, based on the interaction timestamps in the original information flow, uses each vector in the initial feature vector set as a node and constructs a time-series graph. The interaction sequence graph is constructed by considering the order of events and the interaction response relationships. Multimodal association weights between nodes are calculated using feature similarity, and the node connection edges are updated using these weights. After improving the graph structure, a multimodal fusion graph is obtained. The violation assessment module inputs the multimodal fusion graph output by the sequence graph construction module into the graph neural network model. After model message passing and node aggregation operations, a global graph embedding vector representing the deep structure of the interaction (including time association, multimodal association, and call isolation characteristics) is output. The cosine similarity between this vector and each vector in the preset violation pattern vector library is calculated and the score is recorded. If there is a case where the score is higher than the preset threshold, the corresponding violation pattern category is matched to determine the violation type. The confidence score is calculated by combining the violation pattern priority and the similarity score. Finally, an interaction risk assessment conclusion containing the violation type and confidence score is generated.

[0097] This embodiment adapts to the core requirements of real-time interactive scenarios through multi-module collaboration: On the one hand, the information flow acquisition module ensures no omission of original multimodal data, while the multimodal feature extraction module transforms unstructured data into structured vectors, providing a high-quality data foundation for subsequent processing; on the other hand, the time sequence graph construction module combines timestamps and multimodal association weights to enable the multimodal fusion graph to accurately reflect the temporal logic and feature associations of the interaction, helping the graph neural network model to deeply mine the deep information of the interaction; at the same time, the violation assessment module improves the accuracy of violation type determination and the interpretability of results through vector similarity matching and confidence calculation. The division of labor and cooperation among the modules not only achieves comprehensive coverage and effective fusion of multimodal data, but also ensures the real-time and accuracy of the detection process, providing clear and reliable basis for violation intervention for the interactive platform management, and adapting to the compliance detection requirements of high-concurrency real-time interactive scenarios.

[0098] In one embodiment, this application also provides a method for adapting to real-time violation detection in a call center system, which may include:

[0099] During system deployment, the information flow acquisition module connects to the voice gateway and text interaction interface of the call center system to collect all interaction data between customer service representatives and users in real time. At the same time, it connects to the DataStream API of the real-time streaming engine Flink to transmit all interaction data between customer service representatives and users (including user-inputted text messages, customer service replies, audio recordings of both parties' voice calls, screenshots of questions sent by users, and millisecond-level timestamps) to Flink in real time as streaming data. Flink constructs a processing flow based on event-time semantics, uses watermarks (with a 1-second delay) to handle data out-of-order processing, performs key-by partitioning by call ID to achieve state isolation, and distributes the partitioned streaming data to the multimodal feature extraction module.

[0100] The multimodal feature extraction module first mirrors the voice call recording segment via a media server, then directly branches the mirrored voice stream into specialized engines for processing: one branch connects to the ASR engine for real-time transcription to output transcribed text, while the other branch connects to the MFCC engine to extract acoustic features such as pitch and speech rate and output speech acoustic feature vectors. Subsequently, the transcribed text output by ASR and the speech acoustic feature vectors output by MFCC are fed into the real-time streaming engine Flink as streaming data. Flink performs key-by partitioning by call ID to associate the same session data. Simultaneously, the module uses a BERT pre-trained model to extract semantic vectors from the original text sequence and the aforementioned transcribed text, and uses a ResNet model to analyze screenshots to generate visual content vectors. Finally, the partitioned speech acoustic feature vectors, extracted text semantic vectors, and image visual content vectors are aggregated to form an initial feature vector set, which is then uniformly stored in Redis in the format of "Key = Call ID, Value = Initial Feature Vector Set".

[0101] The sequence graph construction module reads the initial feature vector set for the corresponding call from Flink's Redis client (this set is stored in Redis with "call ID" as the key and includes text semantic vectors, speech acoustic feature vectors, and image visual content vectors). Based on the interaction behavior timestamps bound to each feature vector, the vectors are used as nodes to construct an interaction sequence graph in chronological order of the call. Simultaneously, node clusters are partitioned by call ID to achieve information isolation between different call sessions. Then, the multimodal association weight formula is used to calculate the association weights between adjacent nodes and update the connection edges. For key semantic vectors such as "complaint" and "supervision" that appear across time periods in the same call, additional association weights are calculated and new node connection edges are added, ultimately forming a multimodal fusion graph.

[0102] The violation assessment module first inputs the multimodal fusion graph into the GAT graph neural network, and outputs a global graph embedding vector representing the deep structure of the interaction after message passing and node aggregation. Then, it simultaneously performs two core judgment operations: first, it calls a preset violation pattern vector library (containing standard speech acoustic vectors and image visual vectors, corresponding to non-textual modal standard features of 200 violation scenarios such as "complaint & CBRC" and "insulting keyword combinations") stored independently in a vector database (or Redis), calculates the cosine similarity between the global graph embedding vector and each violation pattern vector in the library, and records the score; second, it calls the rule engine Drools (whose rules are defined using the DRL language, containing only text violation matching logic, such as the logical combination of "complaint" and "CBRC," with each rule associated with a priority Salience, and supports dynamic rule updates via KieBase without requiring a system restart), performs violation feature matching on the text sequence and speech-to-text, and outputs a text modality score. If a vector similarity score exceeds a preset threshold (e.g., 0.85 > 0.8), and Drools matches the corresponding text violation rule, the results of both are combined to match the "financial regulatory related complaint violation" category (this rule has a Salience of 100 and the highest priority); subsequently, the confidence formula (assuming...) is applied... =0.4、 =0.6, the normalization priority of the rule Pnorm=1) calculates the confidence level (0.4×1+0.6×0.85)×100%=91%; finally, a risk assessment conclusion labeled "Violation type: financial regulatory related complaint violation, confidence level: 91%, call ID: XXX" is generated. Flink pushes the conclusion to the call quality inspection platform in real time. At the same time, the Drools engine captures the afterActivationFired event when the rule is triggered through the pre-registered AgendaEventListener, extracts information such as the triggering rule ID, call ID, and matching text content, and writes it to the log system (such as ELK) in a preset format.

[0103] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0104] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the real-time interactive violation detection method as described above.

[0105] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0106] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0107] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A real-time interactive violation detection method, characterized in that, The method includes: Obtain the complete raw information stream of the online interactive session; the raw information stream includes user text sequences, voice signals, image files, and timestamps of interactive behaviors; Semantic vectors, acoustic features, and visual content descriptions are extracted from the text sequence, speech signal, and image file in the original information stream to generate an initial feature vector set. Based on the interaction behavior timestamps, the vectors in the initial feature vector set are used as nodes to construct an interaction time sequence graph. Based on the interaction time sequence graph, the multimodal association weights are calculated and the node connection edges are updated to obtain a multimodal fusion graph. The multimodal fusion graph is input into the graph neural network model, which outputs a global graph embedding vector. The similarity score between the global graph embedding vector and the preset violation pattern vector library is calculated, and the violation type of the current interaction is determined, generating an interaction risk assessment conclusion.

2. The method according to claim 1, characterized in that, The process involves extracting semantic vectors, acoustic features, and visual content descriptions from the text sequence, speech signal, and image file in the original information stream, respectively, to generate an initial feature vector set, including: The speech signal in the original information stream is processed in real time to convert the speech signal into text data, thus obtaining speech-transcribed text; For the text sequence and speech-to-text in the original information stream, text semantic information is extracted by text feature extraction method to generate corresponding text semantic vectors; Acoustic attributes, including pitch, speech rate, and audio spectrum, are extracted from the untranscribed speech signal in the original information stream to obtain a speech acoustic feature vector; The image files in the original information stream are analyzed using image recognition technology to identify visual elements in the images, generate visual information describing the image content, and obtain an image visual content vector. The text semantic vectors, speech acoustic feature vectors, and image visual content vectors are summarized and integrated to form an initial feature vector set covering multimodal information.

3. The method according to claim 1, characterized in that, The process of constructing an interaction time sequence graph by using vectors from the initial feature vector set as nodes based on the interaction behavior timestamps, calculating multimodal association weights based on the interaction time sequence graph and updating node connection edges to obtain a multimodal fusion graph includes: Based on the timestamp of the interaction behavior, each feature vector in the initial feature vector set is bound to the corresponding time information to obtain a multimodal feature vector group with time stamp; Using the multimodal feature vector group as nodes, a directed acyclic interaction time sequence diagram is constructed based on the temporal order of each node and the response relationship in real-time interaction, and different call node clusters are divided in the interaction time sequence diagram structure. For the adjacent nodes in the constructed interaction sequence graph, the feature correlation degree between each adjacent node is calculated through feature similarity to obtain the multimodal correlation weights corresponding to the connecting edges of each node; The multimodal association weights are assigned to the corresponding node connection edges in the interaction sequence graph, replacing the initial default connection strength. Based on the context requirements of real-time interaction, association weights are calculated and node connection edges are added for nodes that are cross-statements, non-adjacent but logically related in the same session, in order to obtain the edge structure of the interaction sequence graph. By integrating the updated edge structure and node cluster, a multimodal fusion graph is finally formed that combines time sequence, multimodal association characteristics, and traffic isolation functions.

4. The method according to claim 3, characterized in that, The multimodal association weights corresponding to the connecting edges of each node are calculated using the following formula: in, Represents a node and nodes Multimodal association weights between them Represents a node ,node Text semantic vectors, Represents a node ,node Speech acoustic vectors, Represents a node ,node Image visual vectors, , , Represents the modal weighting coefficient. Indicates feature similarity calculation, This represents the absolute value of the time difference between two nodes. This represents the time decay coefficient.

5. The method according to claim 1, characterized in that, The process involves inputting the multimodal fusion graph into the graph neural network model, outputting a global graph embedding vector, calculating the similarity score between the global graph embedding vector and a preset violation pattern vector library, determining the violation type of the current interaction, and generating an interaction risk assessment conclusion, including: The multimodal fusion graph is input into a graph neural network model, which outputs a global graph embedding vector representing the deep structure of the current interaction process; the deep structure includes time correlation, multimodal correlation, and traffic isolation characteristics; Calculate the cosine similarity between the global graph embedding vector and each violation pattern vector in the pre-established violation pattern vector library, and record the similarity scores; the violation pattern vector library includes preset keyword detection rules; If any of the similarity scores is higher than the preset threshold, the violation pattern category corresponding to the score is directly matched to determine the violation type of the current interaction; For interactions whose violation types have been preliminarily identified, a confidence score corresponding to the violation type is calculated and generated by combining the priority and similarity score of the corresponding violation pattern. By integrating the violation types and confidence scores, and supplementing the corresponding call identifiers for the interactions, an interaction risk assessment conclusion is generated that includes violation type tags, confidence scores, and call identifiers.

6. The method according to claim 5, characterized in that, The confidence score is calculated using the following formula: in, This represents the confidence score. This indicates the normalized priority of the violation pattern, and classifies rules according to the severity of the violation. This represents the combined similarity score between the global graph embedding vector and the preset violation pattern vector. , , These represent the weight coefficients of text modality, speech modality, and view modality in the overall similarity score, respectively. This represents the text modality score. The Drools rule engine, which uses the DRL language to construct complex matching conditions, performs violation feature matching on text sequences and speech-to-text transcripts, and then outputs the result. The speech modality score is calculated by comparing the cosine similarity between the current interaction's speech acoustic feature vector and a preset speech acoustic standard vector. The view modality score is calculated by comparing the cosine similarity between the visual content vector of the currently interacting image and the preset visual standard vector. Indicates priority weight. This represents the similarity weight.

7. A real-time interactive violation detection system, characterized in that, The system includes: The information flow acquisition module is used to acquire all the original information flow of the online interactive session; the original information flow includes user text sequences, voice signals, image files, and timestamps of interactive behaviors. The feature extraction module is used to extract semantic vectors, acoustic features, and visual content descriptions from the text sequence, speech signal, and image file in the original information stream, respectively, and generate an initial feature vector set. The time sequence graph construction module is used to construct an interaction time sequence graph by using the vectors in the initial feature vector set as nodes based on the interaction behavior timestamps, calculate the multimodal association weights based on the interaction time sequence graph and update the node connection edges to obtain a multimodal fusion graph. The violation assessment module is used to input the multimodal fusion graph into the graph neural network model and output a global graph embedding vector, calculate the similarity score between the global graph embedding vector and a preset violation pattern vector library, determine the violation type of the current interaction, and generate an interaction risk assessment conclusion.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images