Multimodal interaction multimedia call center data processing method and system

Abstract

The invention relates to the technical field of data processing and communication, and discloses a multi-modal interaction multimedia call center data processing method and system, and the method comprises the following steps: collecting multi-modal data in real time, and extracting features to generate observation vectors of a customer and a seat; defining state vectors of clients and seats; constructing an adaptive coupling state space model, and describing dynamic coupling of the states of the two parties through a nonlinear transfer equation; performing off-line learning by using historical data and calibrating model parameters; estimating a current state on line and predicting a future trend through an extended Kalman filtering algorithm; and according to a prediction result, generating an interaction guide strategy including risk early warning, behavior recommendation and system intervention. According to the method, the bidirectional coupling adaptive coupling state space model of the customer and the seat is constructed, so that the interactive evolution trend can be accurately predicted, the active guide strategy is generated, and the defect of passive response in the prior art is overcome.

Description

Technical Field

[0001] The present invention relates to the field of data processing and communication technology, and in particular to a multi-modal interactive multimedia call center data processing method and system. Background Art

[0002] With the advancement of information technology, modern call centers have evolved from traditional voice interaction models to multimedia contact centers that integrate voice, text, video, and web pages. In this complex interactive environment, the dimensions of information exchanged between customers and agents have increased dramatically, providing an unprecedented data foundation for deeply understanding the interaction process and improving service quality and operational efficiency. How to effectively process and analyze this massive amount of multimodal data to achieve real-time perception, accurate prediction, and intelligent guidance of interaction status has become a key technical challenge that needs to be addressed in this field.

[0003] Various existing technical solutions have been applied to multimedia call center data processing. For example, automatic speech recognition (ASR) technology is used to convert call content into text; natural language processing (NLP), particularly sentiment analysis models, is used to determine the emotional polarity of customer text or speech; and some solutions also incorporate computer vision technology to assist in identifying customer emotions by analyzing facial expressions during video calls. Based on these analysis results, some systems can automatically perform call quality checks, cluster hot issues, and display knowledge base articles and recommended standard scripts in real time on the agent workstation to assist agents in providing service.

[0004] Although existing technologies have improved the ability to perceive customer status to a certain extent by introducing methods such as multimodal analysis, they still have some shortcomings when dealing with complex two-way interactive dynamics:

[0005] First, existing technologies generally suffer from the limitation of a "one-way perspective." Model construction and analysis focus almost entirely on the customer, aiming to understand their intent and emotions. However, agents are often viewed as stable, undifferentiated, ideal performers, overlooking their own cognitive load and emotional stress. This asymmetric modeling is prevalent because, in traditional thinking, solving customer problems is the primary goal. Agent status is considered part of "human resources" rather than "system modeling," thus technically simplifying the problem. However, interactions are two-way, and agent status is a key variable affecting service quality. A cognitively overloaded or emotionally stressed agent is a significant contributor to service failure. Existing technologies systematically lack understanding of these core risk factors, resulting in a one-sided understanding of the interaction process.

[0006] Secondly, existing analytical models are typically "decoupled and static." They tend to use a series of independent models to handle different single tasks, such as a sentiment classifier and an intent recognizer. They lack a unified dynamic framework that can integrate this scattered state information and reveal its inherent causal relationships. The system may be able to detect the deterioration of customer sentiment, but it cannot quantitatively attribute it to a specific behavior of the agent, nor can it capture the impact of the customer's negative emotions on the agent's cognitive load. This decoupled design stems from the complexity of constructing a unified coupled model. It not only requires a more sophisticated mathematical framework to describe temporal dependencies and interactions, but also places higher demands on data labeling. Therefore, existing analysis remains at the surface level of phenomenon correlation and struggles to delve into the inherent driving mechanisms of interaction.

[0007] Finally, the assistance and intervention measures provided by existing technologies are inherently reactive. Their triggering logic often relies on identifying clear, delayed risk signals, such as a customer uttering a keyword like "complaint" or a sentiment score reaching a certain extreme threshold. The fundamental reason for this approach is that the underlying models are often static classifiers designed to determine "the current state" but lack the ability to predict evolving trends. Summary of the Invention

[0008] In response to the shortcomings of the existing technology, the present invention provides a multimodal interactive multimedia call center data processing method and system, which solves the problem that the existing technology uses a one-way static model analysis, fails to reveal the two-way dynamic coupling mechanism of customer and agent status, and thus cannot accurately predict interaction trends and achieve active guidance.

[0009] To achieve the above objectives, the present invention is implemented through the following technical solutions: A multimodal interactive multimedia call center data processing method, comprising the following steps:

[0010] S1. Real-time collection of voice, text, and video interaction data between customers and agents. Extracting the client's voice acoustic features, text sentiment polarity features, and facial action unit features, as well as the agent's operational behavior features. The system then generates customer observation vectors and agent observation vectors.

[0011] S2. Define a customer state vector and an agent state vector. The customer state vector includes four dimensions: problem-solving progress, emotional valence, emotional arousal, and willingness to cooperate. The agent state vector includes three dimensions: cognitive load index, emotional stress level, and process compliance.

[0012] S3. Based on the customer observation vector, the agent observation vector, the customer state vector, and the agent state vector, construct an adaptive coupled state space model including an observation equation and a state transition equation, wherein:

[0013] The observation equation maps the customer observation vector and the agent observation vector into a linear combination of the customer state vector and the agent state vector respectively;

[0014] The state transfer equation includes a customer state transfer equation and an agent state transfer equation. The customer state transfer equation is a nonlinear equation and the transfer matrix corresponding to the customer state transfer equation changes dynamically with the customer state. The output of the agent state transfer equation is coupled to the control matrix of the customer state transfer equation.

[0015] S4. Using historical interaction data, offline learning is performed on the parameters of the observation equation and the state transfer equation to obtain a calibrated adaptive coupled state-space model;

[0016] S5. Inputting the real-time generated customer observation vector and agent observation vector into the calibrated adaptive coupled state space model, and online estimating the current customer state vector and agent state vector using an extended Kalman filter algorithm, predicting the future multi-step state evolution trend, and generating a future state prediction result;

[0017] S6. Generate an interactive guidance strategy including risk warning, behavior recommendation and system intervention instructions based on the current customer state vector, the agent state vector and the future state prediction result.

[0018] Preferably, in step S1, the step of collecting voice, text and video interaction data between the customer and the agent in real time includes:

[0019] The voice endpoint detection algorithm is used to segment the voice streams of the customer and the agent, and the fundamental frequency, Mel-frequency cepstral coefficients, and short-time energy of the customer's voice segment are extracted as voice acoustic features.

[0020] Using a pre-trained sentiment analysis model to score the sentiment polarity of customer texts to obtain the sentiment polarity features of the texts, wherein the sentiment analysis model is based on the BERT architecture and is obtained after fine-tuning on customer service dialogue corpus;

[0021] Extracting facial action unit features from the client video stream using a facial action coding system, wherein the facial action unit features include corrugator muscle activity, mouth corner upward amplitude, and eyelid closure frequency;

[0022] Statistics are collected on the number of work order operations, knowledge base call delays, and speech deviations performed by agents within a preset time window, and operational behavior characteristics are generated on the agent side.

[0023] Preferably, in step S1, the steps of extracting the voice acoustic features, text emotion polarity features, and facial action unit features of the client, as well as the operation behavior features of the agent end, and generating the customer observation vector and the agent observation vector respectively include:

[0024] Perform endpoint detection on the customer's voice stream, segment the valid voice segments, and extract the fundamental frequency, Mel-frequency cepstral coefficients, and short-time energy as voice acoustic features;

[0025] Use a text sentiment analysis model based on the Transformer architecture to score the sentiment polarity of customer texts and output a continuous value of the text sentiment polarity feature in the range of [-1, 1];

[0026] The facial action coding system analyzes the customer video stream and extracts the intensity values ​​of frowning, mouth corner raising, and blinking to obtain video features and facial action unit features respectively;

[0027] Statistics of agents in time window T w The number of work order submissions, knowledge base query response time and word deviation are used to generate the agent's operational behavior characteristics;

[0028] The client's speech acoustic features, the continuous value of the text emotional polarity features, the video features and the facial action unit features are spliced ​​according to the preset weights to obtain the customer observation vector O s (t), encode the agent's operational behavior characteristics into the agent observation vector O c (t), where O s (t) is the customer observation vector at time t; c (t) is the seat observation vector at time t.

[0029] Preferably, in step S2, the step of defining the customer state vector and the agent state vector includes: dynamically evaluating the trigger status of preset key nodes in the conversation between the customer and the agent, obtaining the weight coefficient and trigger time of each key node, and determining the problem solving progress of the customer state vector of each key node through the weight coefficient, trigger time, and Pearson correlation coefficient regression model of each key node, wherein the Pearson correlation coefficient regression model is:

[0030]

[0031] Where s1(t) is the quantitative value of the problem solving progress at trigger time t, N is the total number of key nodes in the predefined dialogue process, i is the index of the key node, 1≤i≤N, N is a positive integer greater than 1, and w i is the weight coefficient of the i-th key node, I 节点i已触发 is the indicator function of the i-th key node;

[0032] Obtaining the emotional valence of the customer state vector by weighted fusion of a speech emotion polarity score obtained based on a speech emotion recognition model and an emotional polarity score obtained based on a text emotion analysis model;

[0033] Based on the emotional arousal calculation model, the emotional arousal of the customer state vector is obtained by calculating the linear combination of the voice fundamental frequency variance and the blink frequency of the facial action unit. The emotional arousal calculation model is:

[0034]

[0035] Where s3(t) is the estimated value of the emotional arousal of the customer state vector at trigger time t, which is a dimension of the customer state vector and is used to measure the degree of excitement or calmness of the customer's emotions. α is the weight coefficient of the voice fundamental frequency variance, which is used to calibrate the contribution of the fundamental frequency variance to the arousal. is the fundamental frequency variance of the voice, which is used to quantify the fluctuation of the customer's voice tone. β is the weight coefficient of the blink frequency, which is used to calibrate the contribution of the blink frequency to the arousal level. blink The blink frequency per unit time is obtained by analyzing the customer's facial video stream;

[0036] Logistic regression prediction is performed based on customer response delay, negative word frequency, and turn-taking number to obtain the cooperation willingness dimension of the customer state vector.

[0037] Preferably, in step S2, the step of defining the agent state vector further includes:

[0038] The agent state vector includes three dimensions: cognitive load index, emotional stress level and process compliance

[0039] A cognitive load index of the agent state vector is obtained by performing multimodal fusion calculation based on the speaking rate fluctuation rate of the agent's speech, the operation sequence entropy, and the pupil focus collected by the eye tracker;

[0040] The emotional stress level of the agent state vector is obtained by jointly evaluating the agent's voice jitter index, skin conductance sensor data, and speech repetition rate;

[0041] The process compliance degree of the agent state vector is obtained by dynamically updating according to the edit distance between the agent operation and the preset standard process and the number of key node omissions.

[0042] Preferably, in step S3, the step of constructing an adaptive coupled state-space model including an observation equation and a state transfer equation includes:

[0043] The observation equation and state transfer equation constituting the adaptive coupled state space model are respectively:

[0044]

[0045] Where, O s (t) is the customer observation vector at time t, which is composed of speech, text, and visual features; H s is the customer state observation matrix, the customer state observation matrix H s The non-zero element positions of are determined by L1 regularization constraints; S(t) is the customer state vector at time t; v s (t) is the customer observation noise at time t; c (t) is the agent observation vector at time t, which is composed of operation behavior characteristics; H c is the seat state observation matrix; C(t) is the seat state vector at time t; v c (t) is the agent observation noise at time t; A(S(t)) is the nonlinear state transfer matrix; B(S(t), C(t)) is the coupling control matrix; U(t) is the agent behavior vector, encoding the speech type and operation instruction; w s (t) is the customer state process noise; D is the agent state inertia matrix, and the diagonal elements are constrained by ridge regression; E is the customer observation influence matrix; tanh(·) is the hyperbolic tangent activation function; w c (t) is the noise of the seat state process; C(t+1) is the seat state vector at time (t+1); O s (t) is the customer observation vector at time t.

[0046] Preferably, in step S4, the step of using historical interaction data to perform offline learning on the parameters of the observation equation and the state transition equation includes:

[0047] S41, constructing a loss function, the formula of the loss function is:

[0048]

[0049] Where λ is the trade-off coefficient; is the total loss value, which is used to measure the deviation between the model prediction and the true observation; s (t) is the customer observation vector at time t, which is composed of speech, text, and visual features; H s is the customer state observation matrix, which is used to map the hidden state to the observation space; S(t) is the customer state vector at time t; A(S(t)) is the nonlinear state transfer matrix; is the total time step; S(t+1) is the true value of the customer state vector at time t+1;

[0050] S42, based on the loss function and historical interaction data, fix the customer state observation matrix H s and the nonlinear state transfer matrix A(S(t)), and estimate the hidden state sequence by the extended Kalman smoothing algorithm

[0051] S43, based on the observation equation and state transfer equation of the adaptive coupled state space model, fix the hidden state sequence And the seat state observation matrix H c , coupling control matrix B(S(t), C(t)), agent state inertia matrix D and customer observation influence matrix E, update the parameters of the adaptive coupling state space model;

[0052] S44 , repeating steps S42 and S43 to iteratively update the parameters of the adaptive coupled state-space model to obtain a calibrated adaptive coupled state-space model.

[0053] Preferably, in step S5, the steps of inputting the real-time generated customer observation vector and agent observation vector into the calibrated adaptive coupled state space model and online estimating the current customer state vector and agent state vector by using the extended Kalman filter algorithm include: state prediction, covariance prediction, Kalman gain calculation and state update:

[0054] Based on the calibrated nonlinear state transfer equation and state prediction formula, the customer state at the current time t is predicted a priori to obtain the result of the a priori estimation of the customer state at time t. The state prediction formula is:

[0055]

[0056] Where, is the prior estimate of the customer status at time t; is the posterior estimated state based on time t-1 Computed nonlinear transfer matrix; is the posterior estimate of the customer state at time t-1, obtained by Kalman filtering update; is the coupling control matrix; U(t-1) is the agent behavior vector at time (t-1); is the posterior estimate of the agent status at time (t-1);

[0057] The result of the prior estimation of the customer status at time t is verified based on the covariance prediction formula to obtain the prior estimation covariance matrix at time t. The covariance prediction formula is:

[0058]

[0059] Where P(t|t-1) is the prior estimated covariance matrix at time t; J A For A(S(t)) The Jacobian matrix at t-1; P(t-1|t-1) is the posterior estimated covariance matrix at time (t-1); O s is the client state process noise covariance matrix; T is the total time step;

[0060] When the system receives the customer observation vector O at time t s After (t), the Kalman gain matrix at time t is obtained based on the prior estimation covariance matrix P(t|t-1), the customer state observation matrix, the customer observation noise covariance matrix and the Kalman gain calculation formula, and the posterior state estimate at time t is updated based on the posterior state update formula and the customer state prior estimate at time t to obtain the Kalman gain matrix K(t) at time t, wherein the Kalman gain calculation formula is:

[0061]

[0062] Where K(t) is the Kalman gain matrix at time t; H s is the customer state observation matrix; R s is the client observation noise covariance matrix;

[0063] The posterior state update formula is:

[0064]

[0065] Where, is the posterior state estimation vector at time t; is the prior estimate of the customer status at time t; O s (t) is the customer observation vector at time t; is the predicted observation value based on the prior state estimate.

[0066] Preferably, in step S6, the step of generating an interactive guidance strategy including risk warning, behavior recommendation, and system intervention instruction based on the current customer state vector, the agent state vector, and the future state prediction result includes:

[0067] When the estimated value of customer emotional valence <The preset value A of the estimated value of the customer's emotional valence, and the change rate of the customer's emotional valence < preset value B seconds, a high-risk warning is triggered, where: is the estimated value of the customer's emotional valence at time t; is the estimated value of the customer's emotional valence at time (t-1), and Δt is the sampling time interval;

[0068] Based on solving the constrained programming model, the behavior recommendation at time (t+1) is generated by solving the constrained quadratic programming problem. The formula for solving the constrained programming model is:

[0069]

[0070] Where U(t) is the agent behavior vector to be optimized; It is the set of agent operations allowed in the current business scenario; S target is the target customer state vector, which is preset by business rules; is the client state vector at time (t+1) predicted by the adaptive coupled state space model; S target is the target customer state vector; st is the discrete time step; ||·|| is the Euclidean norm;

[0071] When the agent's cognitive load index >The preset value C of the agent cognitive load index and the customer problem solving progress When the customer problem solving progress reaches the preset value D, the system intervention instruction is triggered and the work order is transferred to the senior support team, where: is the estimated value of the agent's cognitive load index at time t; is the estimated progress of solving customer problems at time t.

[0072] The present invention also provides a multi-modal interactive multimedia call center data processing system, comprising:

[0073] The data processing module is used to collect voice, text, and video interaction data between customers and agents in real time, extract the client's voice acoustic features, text emotion polarity features, and facial action unit features, as well as the agent's operational behavior features, and generate customer observation vectors and agent observation vectors respectively;

[0074] A model training module, configured to perform offline learning on the parameters of the observation equation and the state transfer equation using historical interaction data to obtain a calibrated adaptive coupled state-space model;

[0075] An online state estimation module is used to input the customer observation vector and agent observation vector generated in real time by the data processing module into the calibrated adaptive coupled state space model obtained by the model training module, and estimate the current customer state vector and agent state vector online through the extended Kalman filter algorithm, predict the future multi-step state evolution trend, and generate future state prediction results;

[0076] A strategy generation module is used to generate an interactive guidance strategy including risk warning, behavior recommendation and system intervention instructions based on the current customer state vector, the agent state vector and the future state prediction result generated by the online state estimation module.

[0077] The present invention provides a multi-modal interactive multimedia call center data processing method and system.

[0078] It has the following beneficial effects:

[0079] 1. This invention constructs an adaptively coupled state-space model of bidirectional customer and agent data. This model not only depicts the customer's state in multiple dimensions, but also incorporates the agent's cognitive and emotional states into a unified analytical framework, achieving a comprehensive and synchronized perception of the states of both parties in the interaction. This dual-perspective modeling approach more accurately and completely captures the entire multimedia call center interaction process, providing a solid data foundation for subsequent precise prediction and guidance.

[0080] 2. The state transition equation constructed in this invention explicitly establishes the dynamic coupling relationship between the customer state and the agent state and behavior. In particular, through the nonlinear state transition matrix and coupling control matrix, it describes the inherent mechanism of mutual influence between the two parties during the interaction process. This allows insight into causal chains and feedback loops within complex interaction phenomena, providing in-depth analytical capabilities for fundamentally understanding and improving interaction quality.

[0081] 3. This invention utilizes the extended Kalman filter algorithm to perform online inference on the calibrated adaptive coupled state-space model, effectively predicting the future multi-step evolution of interaction states. This forward-looking predictive capability enables the system to move beyond passive real-time monitoring to proactive risk prevention and opportunity discovery. By generating high-risk warnings before service quality deterioration trends become apparent, the system provides agents with a valuable window for intervention, shifting from post-remediation to pre-emptive prevention.

[0082] 4. This invention structures the generation of interaction guidance strategies as a constrained optimization problem, dynamically solving for optimal agent behavior recommendations based on real-time predicted status and pre-set business objectives. This model-based decision-making approach offers greater flexibility and adaptability than traditional methods that rely on fixed rule bases. It can generate personalized, context-aware guidance strategies for ever-changing interaction scenarios, effectively improving agent efficiency and ultimately customer satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 Schematic diagram of the method flow of the present invention;

[0084] Figure 2Schematic diagram of the operation flow of the adaptive coupled state-space model of the present invention;

[0085] Figure 3 A schematic diagram of online state estimation and interactive guidance strategy generation for the present invention;

[0086] Figure 4 This is a system structure diagram of the present invention. DETAILED DESCRIPTION

[0087] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the specification of the present invention. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0088] Please see the attached Figure 1 -Attached Figure 3 The embodiment of the present invention provides a multi-modal interactive multimedia call center data processing method, comprising the following steps:

[0089] S1. Real-time collection of voice, text, and video interaction data between customers and agents. Extracting the client's voice acoustic features, text sentiment polarity features, and facial action unit features, as well as the agent's operational behavior features. The system then generates customer observation vectors and agent observation vectors.

[0090] In this embodiment, the real-time collection and feature extraction of interactive data is achieved through the following technical solutions:

[0091] Data processing modules deployed at customer terminals and agent workstations capture speech acoustic features, text emotion polarity features, and facial action unit features. Dual-channel independent acquisition technology is used to capture audio signals from both customers and agents. A speech endpoint detection algorithm based on energy thresholds is used to segment valid speech segments. The silence threshold is set at -40dB to eliminate ambient noise interference. Text streams are intercepted in real time by monitoring the online chat system's message interface, and the original conversation content is stored in UTF-8 encoding. Video streams capture customer facial images at a rate of at least 25 frames per second, with a resolution of 1280×720 pixels. Transmission is compressed using H.264 encoding to reduce bandwidth usage.

[0092] Client-side speech acoustic feature extraction involves the following steps: For segmented customer speech segments, the median fundamental frequency is calculated to characterize speech intonation; 13-dimensional Mel-Frequency Cepstral Coefficients (MFCCs) are extracted to describe the speech spectrum, with an analysis window of 25ms and a frame shift of 10ms; and the short-term energy of each frame of speech signal is calculated to quantify changes in speech intensity. Text sentiment polarity analysis utilizes a pre-trained model based on the Transformer architecture. Fine-tuned on a domain corpus containing 500,000 customer service conversations, it outputs a continuous sentiment score in the range [-1, 1], where negative values ​​indicate negative sentiment and positive values ​​indicate positive sentiment.

[0093] Customers' facial movement features are analyzed using the Facial Action Coding System (FACS). Specifically, the following action unit strength values ​​are extracted: Corrugator supercilii muscle activity (AU4) reflects customer dissatisfaction, quantified by the contraction of the glabellar muscles; mouth corner lift (AU12) represents positive emotion, calculated by the displacement of key points at the corners of the mouth; and eyelid closure frequency (AU45) is used to assess psychological stress levels by counting the number of blinks per unit time. These action unit strength values ​​are obtained by tracking the coordinate changes of facial key points in real time using a computer vision algorithm.

[0094] The extraction of agent-side operational behavior features includes: counting the number of submission or modification operations in the work order system within a preset time window to reflect workload intensity; recording the time difference between knowledge base query requests and result returns to assess business proficiency; calculating the cosine similarity between the current script and the standard process template, and mapping the text to a 768-dimensional vector space using the Sentence-BERT model for comparison. The deviation calculation formula is:

[0095] 1-cos(v current ,v standard );

[0096] Where, v current is the semantic vector of the current discourse; v standard is the semantic vector of the standard discourse; cos(·, ·) is the cosine similarity function.

[0097] The customer observation vector is generated through weighted concatenation of multimodal features, including speech acoustic features (fundamental frequency, MFCC, short-term energy), text sentiment score, and facial action unit intensity values. Each feature dimension is normalized according to preset weights, with speech features weighted 40%, text sentiment 30%, and visual features 30%, thereby balancing the contributions of different modalities to state estimation. The agent observation vector is composed of standardized values ​​for three dimensions: work order operation frequency, knowledge base response latency, and speech deviation. Min-max normalization is used to map each metric to the [0, 1] range and then directly concatenated.

[0098] Among them, the voice endpoint detection algorithm distinguishes valid speech from background noise, ensuring the accuracy of acoustic feature extraction. The domain-tuned sentiment analysis model better captures emotional expression characteristics in customer service scenarios than general-purpose models. The selection of facial action units (AU4, AU12, and AU45) is based on psychological research confirming their high correlation with customer emotional states. The setting of agent operational behavior indicators (work order operation, knowledge base delay, and word deviation) comprehensively reflects their work efficiency and process adherence. The weight distribution strategy of the observation vector is determined through historical data regression analysis to ensure the optimal fusion of multimodal information.

[0099] S2. Define a customer state vector and an agent state vector. The customer state vector includes four dimensions: problem-solving progress, emotional valence, emotional arousal, and willingness to cooperate. The agent state vector includes three dimensions: cognitive load index, emotional stress level, and process compliance.

[0100] In this embodiment, the definition and calculation method of the customer state vector and the agent state vector are implemented by the following technical solutions:

[0101] The customer state vector consists of four interrelated dimensions, each representing the customer's core state attributes during the interaction. The problem resolution progress dimension is dynamically assessed by triggering states at key nodes preset in the conversation process. Key nodes include identity verification completion, problem type confirmation, solution proposal, and customer confirmation. The weight coefficient for each node is determined by regressing the Pearson correlation coefficient between the node trigger time and the overall resolution time in historical interaction data. The specific calculation formula is:

[0102]

[0103] Where s1(t) is the quantitative value of the problem solving progress at trigger time t, N is the total number of key nodes in the predefined dialogue process, i is the index of the key node, 1≤i≤N, N is a positive integer greater than 1, and w i is the weight coefficient of the i-th key node, I 节点i已触发 is the indicator function of the i-th key node.

[0104] The emotional valence dimension integrates the results of bimodal sentiment analysis of speech and text. Speech emotion recognition uses an acoustic model based on a deep neural network to extract features such as fundamental frequency and spectral centroid, and then outputs a sentiment polarity score. Text sentiment analysis uses a BERT model fine-tuned on customer service corpus, outputting continuous values ​​in the range [-1, 1]. The two are combined with a weight ratio of 0.6:0.4, and the calculation formula is:

[0105] s2(t)=0.6·p voice +0.4·ptex;

[0106] Where s2(t) is the estimated value of the customer's emotional valence at time t, which is a dimension of the customer's state vector and is used to measure the degree of positivity or negativity of the customer's emotions; p voice is the speech emotion polarity score obtained by analyzing the customer's speech stream using the speech emotion recognition model; ptex is the text emotion polarity score obtained by analyzing the customer's input text message using the text emotion analysis model; 0.6 and 0.4 are the fusion weights of the speech modality and text modality, respectively, indicating that in this embodiment, the contribution of speech emotion is considered to be slightly higher than that of text emotion;

[0107] The weight allocation strategy is optimized on the validation set through grid search to maximize the accuracy of emotional state classification.

[0108] The emotional arousal dimension quantifies the customer's physiological activation level and is calculated by a linear combination of the voice fundamental frequency variance and the facial blink frequency. The voice fundamental frequency variance reflects the intensity of intonation fluctuations and is calculated as:

[0109]

[0110] Where, T is the variance of the fundamental frequency of speech in a time window; T is the total number of valid speech frames contained in the time window; t is the index of the speech frame; F0 t is the instantaneous fundamental frequency value of the t-th frame speech signal, in Hertz (Hz); μ F0 is the fundamental frequency value of all speech frames in the time window.

[0111] The blink frequency is converted from the AU45 action unit intensity value extracted by the Facial Action Coding System (FACS). The final arousal is calculated as:

[0112]

[0113] Where s3(t) is the estimated value of the emotional arousal of the customer state vector at trigger time t, which is a dimension of the customer state vector and is used to measure the degree of excitement or calmness of the customer's emotions. α is the weight coefficient of the voice fundamental frequency variance, which is used to calibrate the contribution of the fundamental frequency variance to the arousal. is the fundamental frequency variance of the voice, which is used to quantify the fluctuation of the customer's voice tone. β is the weight coefficient of the blink frequency, which is used to calibrate the contribution of the blink frequency to the arousal level. blink It is the blink frequency per unit time obtained by analyzing the customer's facial video stream.

[0114] The cooperation willingness dimension is predicted using a logistic regression model. Input features include customer response latency, negation word frequency, and turn-taking. Response latency is defined as the median time interval between two consecutive customer statements. Negation word frequency is calculated using a predefined dictionary match statistics. Turn-taking is detected using a timestamp-based turn-taking analysis algorithm. The model output is mapped to the [0, 1] interval using a sigmoid function, representing the probability of cooperation willingness.

[0115] The agent state vector consists of three dimensions, used to assess their work status and performance in real time. The cognitive load index integrates three metrics: speech rate fluctuation, operation sequence entropy, and pupil focus. Speech rate fluctuation is calculated by calculating the standard deviation of the average speaking rate over the last 10 seconds; operation sequence entropy quantifies the randomness of agent operations (such as ticket submission and knowledge base query) based on Shannon entropy theory; and pupil focus is calculated from gaze data collected by an eye tracker, reflecting the degree of attention. The normalized values ​​of these three metrics are combined using a weighted average, with the weight coefficient determined by expert evaluation.

[0116] Emotional stress levels are assessed through a combination of multiple physiological signals and behavioral indicators: the voice jitter index is calculated by the percentage of frames with fundamental frequency jitter exceeding 0.5%; the skin conductance response (GSR) data is normalized using a sliding window and the standard deviation is extracted; and the speech repetition rate counts the number of times the same speech template is used within the last minute. The three are combined in a weighted ratio of 0.5:0.3:0.2, and the calculation formula is:

[0117] c2(t)=0.5·j voice +0.3·σ GSR +0.2·r repeat ;

[0118] Where c2(t) is the estimated value of the agent's emotional stress level at time t, which is a dimension of the agent's state vector and is used to quantify the agent's psychological stress during the interaction process; j voice Jitter Index is calculated by analyzing the agent's voice stream. It usually refers to the percentage of small disturbances in the fundamental frequency period. A higher value indicates a more unstable voice, which may reflect nervousness. GSR is the standard deviation of the Galvanic Skin Response, which is calculated using data collected by physiological sensors (such as wristbands) deployed at the agent end. This value reflects the volatility of skin conductance levels and is a common physiological indicator for measuring emotional arousal and stress. repeatThe repetition rate of speech refers to the number of times or frequency that agents repeatedly use the same or highly similar speech templates within a preset time window (for example, the last minute). This indicator may reflect the rigidity of the agent's thinking or difficulty in coping; 0.5, 0.3, and 0.2 are the fusion weights of each indicator respectively; process compliance dynamically evaluates the degree of deviation between the agent's operation and the standard process: the edit distance algorithm is used to calculate the number of steps that differ between the actual operation sequence and the preset process template, and corrections are made based on the number of key node omissions. The calculation formula is:

[0119]

[0120] Where c3(t) is the quantitative value of the agent process compliance at time t, which is a dimension of the agent state vector and its value range is usually between [0,1].

[0121] d edit is the edit distance, which is calculated by comparing the agent's actual operation sequence with the preset standard business process template. It represents the minimum number of single-step operations (such as insertion, deletion, and replacement) required to convert the actual operation sequence into the standard process sequence; γ is the key node omission penalty coefficient, which is a preset positive real number weight used to increase the penalty for omitting key business links (such as failure to conduct identity authentication or failure to inform important terms); n miss is the total number of key business nodes missed by the agent during the interaction process up to time tt; d max It is a preset maximum tolerance distance or normalization factor, representing the maximum acceptable process deviation in a specific business scenario.

[0122] The design of each dimension of the customer state vector is based on psychological and behavioral theories. Problem-solving progress reflects interaction efficiency, emotional valence and arousal correspond to two orthogonal dimensions of the valence-arousal model, and willingness to cooperate quantifies customer engagement. The multimodal fusion strategy of the agent state vector ensures comprehensive monitoring of cognitive load, emotional stress, and process compliance. The weighting coefficients and calculation methods are optimized through a data-driven approach to avoid bias introduced by subjective assumptions.

[0123] S3. Based on the customer observation vector, the agent observation vector, the customer state vector, and the agent state vector, construct an adaptive coupled state space model including an observation equation and a state transition equation, wherein:

[0124] The observation equation maps the customer observation vector and the agent observation vector into a linear combination of the customer state vector and the agent state vector respectively;

[0125] The state transfer equation includes a customer state transfer equation and an agent state transfer equation. The customer state transfer equation is a nonlinear equation and the transfer matrix corresponding to the customer state transfer equation changes dynamically with the customer state. The output of the agent state transfer equation is coupled to the control matrix of the customer state transfer equation.

[0126] In this embodiment, the construction of the adaptive coupled state-space model is achieved through the following technical solutions:

[0127] Customer observation vector O s (t) and the agent observation vector O c (t) are associated with the hidden state through linear mapping relationships:

[0128]

[0129] Where, O s (t) is the customer observation vector at time t, which is composed of speech, text, and visual features; H s is the customer state observation matrix, and the position of its non-zero elements is determined by the L1 regularization constraint; S(t) is the customer state vector at time t; v s (t) is the customer observation noise at time t; c (t) is the agent observation vector at time t, which is composed of operation behavior characteristics; H c is the seat state observation matrix; C(t) is the seat state vector at time t; v c (t) is the seat observation noise at time t;

[0130] This design makes the observation matrix only retain the mapping of the key dimensions of the state vector, suppressing the interference of redundant features. and The covariance matrix of is initialized by estimating the covariance of historical data samples.

[0131] The dynamic evolution of customer status is modeled as a nonlinear process, and its transfer matrix is ​​adaptively adjusted with the current status:

[0132] S(t+1)=A(S(t))S(t)+B(S(t),C(t))U(t)+w s (t);

[0133] Where A(S(t)) is the nonlinear state transfer matrix; B(S(t), C(t)) is the coupling control matrix; U(t) is the agent behavior vector, encoding the speech type and operation instruction; w s (t) is the client state process noise.

[0134] The evolution of agent status is influenced by both agent inertia and customer observation feedback:

[0135] C(t+1)=DC(t)+Etanh(O s (t))+w c (t);

[0136] Where D is the agent state inertia matrix, and the diagonal elements are constrained by ridge regression; E is the customer observation influence matrix; tanh(·) is the hyperbolic tangent activation function; w c (t) is the noise of the seat state process; C(t+1) is the seat state vector at time t+1; O s (t) is the customer observation vector at time t.

[0137] The nonlinear design of the customer state transition equation captures the dynamic coupling between emotional valence and problem-solving progress. For example, high emotional arousal can lead to a nonlinear acceleration of problem-solving progress. The introduction of the tanh(·) function in the agent state equation effectively limits the magnitude of the customer observation influence, preventing overflow of the state value domain. The bilinear structure of the control matrix B(S(t), C(t)) explicitly encodes the "customer emotion-agent action" interaction logic, which aligns with the psychological mechanisms of behavioral influence in customer service scenarios.

[0138] S4. Using historical interaction data, offline learning is performed on the parameters of the observation equation and the state transfer equation to obtain a calibrated adaptive coupled state-space model;

[0139] In this embodiment, the process of offline learning of the adaptive coupled state-space model parameters using historical interaction data is achieved through the following technical solutions:

[0140] To learn the unknown parameters of an adaptive coupled state-space model from historical interaction data, we construct a comprehensive loss function that aims to balance the model's goodness of fit to the observed data with its adherence to state transition patterns. Specifically, the loss function is defined as the weighted sum of the observation reconstruction error and the state transition prediction error, and its mathematical expression is as follows:

[0141]

[0142] Where λ is the trade-off coefficient; is the total loss value, which is used to measure the deviation between the model prediction and the true observation; s (t) is the customer observation vector at time t, which is composed of speech, text, and visual features; H s is the customer state observation matrix, which is used to map the hidden state to the observation space; S(t) is the customer state vector at time t; A(S(t)) is the nonlinear state transfer matrix; is the total time step; S(t+1) is the true value of the customer state vector at time t+1.

[0143] Because the loss function contains both unknown model parameters and the unknown hidden state sequence (S(t), C(t)), direct joint optimization is difficult to solve. Therefore, this paper adopts the idea of ​​alternating direction multipliers (ADMM) to decompose the complex joint optimization problem into two easier-to-solve subproblems that are executed alternately until the model parameters converge.

[0144] In this step, all model parameters of the current iteration round are temporarily fixed, including the observation matrix H s 、H c And all parameter matrices A(·), B(·), D, E in the state transfer equation. At this time, the optimization goal is based on the given model parameters and the complete observation sequence Infer the most likely hidden state sequence

[0145] Since the state transition equation of the present invention contains nonlinear terms, the standard Kalman smoothing algorithm is no longer applicable. Therefore, it is preferred to use the Extended Kalman Smoother (EKS) algorithm. The algorithm first performs a forward pass (i.e., the standard extended Kalman filter) to obtain the filtered state estimate for each time step; then, a backward pass is performed, using the time from t+1 to The forward filtering result is modified by all future observation information, thus obtaining a smoothed state estimate of the entire time series. This smoothed estimate has higher accuracy than the filtered value using only historical information.

[0146] Step 2: Model parameter update (M-Step)

[0147] In this step, the hidden state sequence estimated in the previous step is The loss function is then transformed into an optimization problem involving only the model parameters. The goal is to find a new set of model parameters such that, under this set of parameters, the observations and state transitions generated by the model best match the "real" observations and estimated state sequences.

[0148] Taking into account the large number of parameters to be optimized and the non-convex objective function, this embodiment uses the Quasi-Newton Method to update the model parameters. Specifically, it is preferred to use the L-BFGS (Limited-memory Broyden–Fletcher–Goldfarb–Shanno) algorithm, which approximates the inverse of the Hessian matrix by storing the gradient change information of the most recent iterations, thereby significantly reducing the memory overhead while ensuring a faster convergence speed. It is particularly suitable for scenarios with larger parameter scales in the present invention. Convergence and termination The above two steps (hidden state sequence estimation and model parameter update) are performed alternately and iteratively. After each round of iteration, the total loss function is calculated. The optimization process terminates when the rate of decrease of the loss function between two consecutive iterations is less than the preset convergence threshold, or when the preset maximum number of iterations is reached. The model parameters obtained at this time are the calibrated adaptive coupled state-space model parameters and can be used for subsequent online state estimation.

[0149] S5. Using historical interaction data, offline learning is performed on the parameters of the observation equation and the state transfer equation to obtain a calibrated adaptive coupled state-space model;

[0150] In this embodiment, the process of inputting the real-time observation vector into the calibrated adaptive coupled state-space model to online estimate and predict the interaction state is achieved through the following technical solutions:

[0151] After the model parameters are calibrated offline in step S4, the system enters the online operation phase. For each new time step t, the system collects and generates the customer observation vector O in real time. s (t) and the agent observation vector O C (t) is used as input and recursive state estimation is performed using the Extended Kalman Filter (EKF) algorithm. This algorithm is essentially a closed-loop prediction-update iterative process that aims to optimally estimate the internal state that cannot be directly observed from noisy observation data in real time.

[0152] Before receiving the observation data at time t, the system first estimates the optimal posterior state at time t-1. and As well as the action vector U(t-1) taken by the agent at the previous moment, the customer state at the current moment t is predicted a priori. This prediction process utilizes a calibrated nonlinear state transfer equation, whose mathematical expression is:

[0153]

[0154] Where, is the prior estimate of the customer status at time t; is the posterior estimated state based on time t-1 Computed nonlinear transfer matrix; is the posterior estimate of the customer state at time t-1, obtained by Kalman filtering update; is the coupling control matrix; U(t-1) is the agent behavior vector at time (t-1); is the posterior estimate of the agent status at time (t-1).

[0155] Along with state prediction, the system also needs to simultaneously predict the uncertainty of the prior estimate, that is, the propagation of the error covariance matrix. Since the state transfer equation is nonlinear, its uncertainty propagation cannot be simply transferred linearly. Therefore, it is necessary to first calculate the nonlinear transfer function A(·) in Perform a first-order Taylor expansion at the position to obtain its Jacobian matrix J A Then, the Jacobian matrix is ​​used to approximately linearly propagate the posterior error covariance P(t-1|t-1) of the previous moment, and the inherent process noise covariance O of the state transition process itself is superimposed. s . Its mathematical expression is:

[0156]

[0157] Where P(t|t-1) is the prior estimated covariance matrix at time t; J A For A(S(t)) The Jacobian matrix at t-1; P(t-1|t-1) is the posterior estimated covariance matrix at time (t-1); O s is the client state process noise covariance matrix; T is the total time step.

[0158] When the system receives the real customer observation vector O at time t s (t), the update phase begins. First, the Kalman gain matrix K(t) needs to be calculated. The core function of this gain matrix is ​​to balance the state prediction value and the new observation value to determine the degree to which the new observation data is trusted. Its calculation formula is:

[0159]

[0160] Where K(t) is the Kalman gain matrix at time t; H s is the customer state observation matrix; R s is the customer observation noise covariance matrix, which reflects the credibility of the observation data itself. sIf the prediction uncertainty P(t|t-1) is large, the value of the Kalman gain K(t) will increase accordingly, which means that the system will be more inclined to adopt new observation information to correct the prediction.

[0161] Finally, the system uses the Kalman gain K(t) and the residual between the true observation and the predicted observation to estimate the prior state Make corrections to obtain the optimal posterior state estimate at the current time t The mathematical expression of the update process is:

[0162]

[0163] Where, is the posterior state estimation vector at time t; is the prior estimate of the customer status at time t; O s (t) is the customer observation vector at time t; is the predicted observation value based on the prior state estimate.

[0164] At the same time, the error covariance matrix at this moment is also updated accordingly to P(t|t)=(IK(t)H s )P(t|t-1), the updated covariance will be used as the input for the next round of prediction, forming a complete recursive filtering cycle.

[0165] Obtaining the optimal state estimate at the current moment After that, the system can further predict the state evolution trend of multiple time steps in the future by recursively applying the state transition equation. For example, the state at time t+k can be predicted by The iterative calculations assume that the future agent behavior U will maintain its current state or follow a pre-set strategy. This multi-step prediction capability provides a decision-making basis for the subsequent generation of proactive interaction guidance strategies.

[0166] S6. Generate an interactive guidance strategy including risk warning, behavior recommendation, and system intervention instructions based on the current customer state vector, the agent state vector, and the future state prediction result;

[0167] In this embodiment, the process of generating an interactive guidance strategy based on the current and predicted state vectors includes three levels of technical solutions: risk warning, behavior recommendation, and system intervention.

[0168] The present invention implements a risk warning mechanism based on real-time state estimation, which aims to promptly identify key risk points in the interaction process that may lead to a decline in customer satisfaction. This mechanism is implemented by continuously monitoring the customer state vector estimate output by the extended Kalman filter algorithm in step S5.

[0169] Preferably, when the system detects the estimated value of the customer's emotional valence When the value of the emotional valence is lower than the preset negative threshold and the rate of change shows a rapid deterioration trend, a high-risk warning will be triggered. The rate of change of the emotional valence is approximated by the first-order backward difference, that is, Where Δt is the sampling time interval.

[0170] The specific trigger logic is:

[0171] when and When the customer's mood is negative, the system determines that the customer's mood is at high risk. This complex condition is designed to capture both the absolute negative state of the customer's mood and its dynamic deterioration trend, effectively avoiding false alarms caused by short-term mood swings and improving the accuracy and timeliness of early warnings. Early warning signals can be presented visually or audibly on the agent's work interface.

[0172] When potential risks are identified or during regular interactions, the present invention can proactively generate optimal behavioral recommendations for agents to guide interactions in a positive direction. These behavioral recommendations are not based on a fixed rule base, but are generated dynamically by solving a constrained quadratic programming problem.

[0173] Its goal is to find an optimal agent behavior vector U(t) within the operating range allowed by the current business scenario, so that the customer state vector at time t+1 predicted by the adaptive coupled state space model is Compared with the preset ideal target state S target The Euclidean distance between them is minimum.

[0174] Its mathematical model is expressed as:

[0175]

[0176] Where U(t) is the agent behavior vector to be optimized; It is the set of agent operations allowed in the current business scenario; S target is the target customer state vector, which is preset by business rules; is the customer state vector at time t+1 predicted by the adaptive coupled state space model; S target is the target customer state vector; t is the discrete time step; ||·|| is the Euclidean norm.

[0177] This optimization problem can be solved within milliseconds using numerical optimization algorithms such as the interior point method, and the optimal solution U * (t) will be translated into specific, actionable behavioral recommendations and pushed to the agents.

[0178] To handle extreme situations that agents find difficult to handle independently, the present invention also includes an automated system intervention mechanism. This mechanism is designed to implement stronger intervention measures beyond conventional recommendations to ensure service quality when the combination of agent and customer status indicates an impasse.

[0179] When the system monitors the estimated value of the agent's cognitive load index If the value is continuously higher than a high threshold, it indicates that the agent may be in a state of information overload or having difficulty handling the current complex problem, and at the same time, the estimated progress of the customer's problem resolution If the value is still below a low threshold, indicating that the interaction has been stagnant for a long time, the system will automatically trigger the work order transfer process;

[0180] The mathematical expression of this condition is:

[0181] Trigger intervention

[0182] Where, is the estimated value of the agent's cognitive load index at time t; is the estimated value of the customer problem solving progress at time t; 0.8 is the preset high threshold of the agent cognitive load index; 0.3 is the preset low threshold of the customer problem solving progress; ∧ is the logical AND operator, indicating that both conditions must be met at the same time for the intervention rule to be triggered.

[0183] Once this condition is met, the system will automatically generate and execute a work order transfer instruction, and through the interface with the back-end call center automated call center dispatch (ACD) system or customer relationship management (CRM) system, seamlessly transfer the current conversation to a more experienced expert agent to achieve timely service escalation and problem resolution.

[0184] The multimodal interactive multimedia call center data processing system described below and the multimodal interactive multimedia call center data processing system method described above may refer to each other.

[0185] Please see the attached Figure 4 The present invention also provides a multi-modal interactive multimedia call center data processing system, comprising:

[0186] The data processing module is used to collect voice, text, and video interaction data between customers and agents in real time, extract the client's voice acoustic features, text emotion polarity features, and facial action unit features, as well as the agent's operational behavior features, and generate customer observation vectors and agent observation vectors respectively;

[0187] A model training module, configured to perform offline learning on the parameters of the observation equation and the state transfer equation using historical interaction data to obtain a calibrated adaptive coupled state-space model;

[0188] An online state estimation module is used to input the customer observation vector and agent observation vector generated in real time by the data processing module into the calibrated adaptive coupled state space model obtained by the model training module, and estimate the current customer state vector and agent state vector online through the extended Kalman filter algorithm, predict the future multi-step state evolution trend, and generate future state prediction results;

[0189] A strategy generation module is used to generate an interactive guidance strategy including risk warning, behavior recommendation and system intervention instructions based on the current customer state vector, the agent state vector and the future state prediction result generated by the online state estimation module.

[0190] The system of this embodiment can be used to execute the above method embodiments, and its principles and technical effects are similar, so they will not be repeated here.

[0191] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal interactive multimedia call center data processing method, characterized in that: The following steps are involved: S1. Real-time collection of voice, text, and video interaction data between customers and agents. Extracting the client's voice acoustic features, text sentiment polarity features, and facial action unit features, as well as the agent's operational behavior features. The system then generates customer observation vectors and agent observation vectors. S2. Define a customer state vector and an agent state vector. The customer state vector includes four dimensions: problem-solving progress, emotional valence, emotional arousal, and willingness to cooperate. The agent state vector includes three dimensions: cognitive load index, emotional stress level, and process compliance. S3. Based on the customer observation vector, the agent observation vector, the customer state vector, and the agent state vector, construct an adaptive coupled state space model including an observation equation and a state transition equation, wherein: The observation equation maps the customer observation vector and the agent observation vector into a linear combination of the customer state vector and the agent state vector respectively; The state transfer equation includes a customer state transfer equation and an agent state transfer equation. The customer state transfer equation is a nonlinear equation and the transfer matrix corresponding to the customer state transfer equation changes dynamically with the customer state. The output of the agent state transfer equation is coupled to the control matrix of the customer state transfer equation. S4. Using historical interaction data, offline learning is performed on the parameters of the observation equation and the state transfer equation to obtain a calibrated adaptive coupled state-space model; S5. Inputting the real-time generated customer observation vector and agent observation vector into the calibrated adaptive coupled state space model, and online estimating the current customer state vector and agent state vector using an extended Kalman filter algorithm, predicting the future multi-step state evolution trend, and generating a future state prediction result; S6. Generate an interactive guidance strategy including risk warning, behavior recommendation and system intervention instructions based on the current customer state vector, the agent state vector and the future state prediction result.

2. A multimodal interactive multimedia call center data processing method according to claim 1, characterized in that: In step S1, the step of collecting the voice, text and video interaction data between the customer and the agent in real time includes: The voice endpoint detection algorithm is used to segment the voice streams of the customer and the agent, and the fundamental frequency, Mel-frequency cepstral coefficients, and short-time energy of the customer's voice segment are extracted as voice acoustic features. Using a pre-trained sentiment analysis model to score the sentiment polarity of customer texts to obtain the sentiment polarity features of the texts, wherein the sentiment analysis model is based on the BERT architecture and is obtained after fine-tuning on customer service dialogue corpus; Extracting facial action unit features from the client video stream using a facial action coding system, wherein the facial action unit features include corrugator muscle activity, mouth corner upward amplitude, and eyelid closure frequency; Statistics are collected on the number of work order operations, knowledge base call delays, and speech deviations performed by agents within a preset time window, and operational behavior characteristics are generated on the agent side.

3. The method for processing multimodal interactive multimedia call center data according to claim 2, characterized in that: In step S1, the steps of extracting the voice acoustic features, text emotion polarity features, and facial action unit features of the client, as well as the operation behavior features of the agent, and generating the customer observation vector and the agent observation vector respectively include: Perform endpoint detection on the customer's voice stream, segment the valid voice segments, and extract the fundamental frequency, Mel-frequency cepstral coefficients, and short-time energy as voice acoustic features; Use a text sentiment analysis model based on the Transformer architecture to score the sentiment polarity of customer texts and output a continuous value of the text sentiment polarity feature in the range of [-1, 1]; The facial action coding system analyzes the customer video stream and extracts the intensity values ​​of frowning, mouth corner raising, and blinking to obtain video features and facial action unit features respectively; Statistics of agents in time window T w The number of work order submissions, knowledge base query response time and word deviation are used to generate the agent's operational behavior characteristics; The client's speech acoustic features, the continuous value of the text emotional polarity features, the video features and the facial action unit features are spliced ​​according to the preset weights to obtain the customer observation vector O s (t), encode the agent's operational behavior characteristics into the agent observation vector O c (t), where O s (t) is the customer observation vector at time t; c (t) is the seat observation vector at time t.

4. The method for processing multimodal interactive multimedia call center data according to claim 3, characterized in that: In step S2, the step of defining the customer state vector and the agent state vector includes: dynamically evaluating the trigger status of preset key nodes in the conversation between the customer and the agent, obtaining the weight coefficient and trigger time of each key node, and determining the problem solving progress of the customer state vector at each key node through the weight coefficient, trigger time, and Pearson correlation coefficient regression model of each key node, wherein the Pearson correlation coefficient regression model is: Where s1(t) is the quantitative value of the problem solving progress at trigger time t, N is the total number of key nodes in the predefined dialogue process, i is the index of the key node, 1≤i≤N, N is a positive integer greater than 1, and w i is the weight coefficient of the i-th key node, I 节点i已触发 is the indicator function of the i-th key node; Obtaining the emotional valence of the customer state vector by weighted fusion of a speech emotion polarity score obtained based on a speech emotion recognition model and an emotional polarity score obtained based on a text emotion analysis model; Based on the emotional arousal calculation model, the emotional arousal of the customer state vector is obtained by calculating the linear combination of the voice fundamental frequency variance and the blink frequency of the facial action unit. The emotional arousal calculation model is: Where s3(t) is the estimated value of the emotional arousal of the customer state vector at trigger time t, which is a dimension of the customer state vector and is used to measure the degree of excitement or calmness of the customer's emotions. α is the weight coefficient of the voice fundamental frequency variance, which is used to calibrate the contribution of the fundamental frequency variance to the arousal. is the fundamental frequency variance of the voice, which is used to quantify the fluctuation of the customer's voice tone. β is the weight coefficient of the blink frequency, which is used to calibrate the contribution of the blink frequency to the arousal level. blink The blink frequency per unit time is obtained by analyzing the customer's facial video stream; Logistic regression prediction is performed based on customer response delay, negative word frequency, and turn-taking number to obtain the cooperation willingness dimension of the customer state vector.

5. The method for processing multimodal interactive multimedia call center data according to claim 4, characterized in that: In step S2, the step of defining the agent state vector further includes: The agent state vector includes three dimensions: cognitive load index, emotional stress level and process compliance A cognitive load index of the agent state vector is obtained by performing multimodal fusion calculation based on the speaking rate fluctuation rate of the agent's speech, the operation sequence entropy, and the pupil focus collected by the eye tracker; The emotional stress level of the agent state vector is obtained by jointly evaluating the agent's voice jitter index, skin conductance sensor data, and speech repetition rate; The process compliance degree of the agent state vector is obtained by dynamically updating according to the edit distance between the agent operation and the preset standard process and the number of key node omissions.

6. The method for processing multimodal interactive multimedia call center data according to claim 1, characterized in that: In step S3, the step of constructing an adaptive coupled state-space model including an observation equation and a state transfer equation includes: The observation equation and state transfer equation constituting the adaptive coupled state space model are respectively: Where, O s (t) is the customer observation vector at time t, which is composed of speech, text, and visual features; H s is the customer state observation matrix, the customer state observation matrix H s The non-zero element positions of are determined by L1 regularization constraints; S(t) is the customer state vector at time t; v s (t) is the customer observation noise at time t; c (t) is the agent observation vector at time t, which is composed of operation behavior characteristics; H c is the seat state observation matrix; C(t) is the seat state vector at time t; v c (t) is the agent observation noise at time t; A(S(t)) is the nonlinear state transfer matrix; B(S(t), C(t)) is the coupling control matrix; U(t) is the agent behavior vector, encoding the speech type and operation instruction; w s (t) is the customer state process noise; D is the agent state inertia matrix, and the diagonal elements are constrained by ridge regression; E is the customer observation influence matrix; tanh(·) is the hyperbolic tangent activation function; w c (t) is the noise of the seat state process; C(t+1) is the seat state vector at time (t+1); O s (t) is the customer observation vector at time t.

7. The method for processing multimodal interactive multimedia call center data according to claim 6, characterized in that: In step S4, the step of using historical interaction data to perform offline learning on the parameters of the observation equation and the state transition equation includes: S41, constructing a loss function, the formula of the loss function is: Where λ is the trade-off coefficient; is the total loss value, which is used to measure the deviation between the model prediction and the true observation; s (t) is the customer observation vector at time t, which is composed of speech, text, and visual features; H s is the customer state observation matrix, which is used to map the hidden state to the observation space; S(t) is the customer state vector at time t; A(S(t)) is the nonlinear state transfer matrix; T is the total time step; S(t+1) is the true value of the customer state vector at time t+1; S42, based on the loss function and historical interaction data, fix the customer state observation matrix H s and the nonlinear state transfer matrix A(S(t)), and estimate the hidden state sequence by the extended Kalman smoothing algorithm S43, based on the observation equation and state transfer equation of the adaptive coupled state space model, fix the hidden state sequence And the seat state observation matrix H c , coupling control matrix B(S(t), C(t)), agent state inertia matrix D and customer observation influence matrix E, update the parameters of the adaptive coupling state space model; S44 , repeating steps S42 and S43 to iteratively update the parameters of the adaptive coupled state-space model to obtain a calibrated adaptive coupled state-space model.

8. The method for processing multimodal interactive multimedia call center data according to claim 7, characterized in that: In step S5, the steps of inputting the real-time generated customer observation vector and agent observation vector into the calibrated adaptive coupled state space model and online estimating the current customer state vector and agent state vector through the extended Kalman filter algorithm include: state prediction, covariance prediction, Kalman gain calculation and state update: Based on the calibrated nonlinear state transfer equation and state prediction formula, the customer state at the current time t is predicted a priori to obtain the result of the a priori estimation of the customer state at time t. The state prediction formula is: Where, is the prior estimate of the customer status at time t; is the posterior estimated state based on time t-1 Computed nonlinear transfer matrix; is the posterior estimate of the customer state at time t-1, obtained by Kalman filtering update; is the coupling control matrix; U(t-1) is the agent behavior vector at time (t-1); is the posterior estimate of the agent status at time (t-1); The result of the prior estimation of the customer status at time t is verified based on the covariance prediction formula to obtain the prior estimation covariance matrix at time t. The covariance prediction formula is: Where P(t|t-1) is the prior estimated covariance matrix at time t; J A For A(S(t)) The Jacobian matrix at t-1; P(t-1|t-1) is the posterior estimated covariance matrix at time (t-1); O s is the client state process noise covariance matrix; T is the total time step; When the system receives the customer observation vector O at time t s After (t), the Kalman gain matrix at time t is obtained based on the prior estimation covariance matrix P(t|t-1), the customer state observation matrix, the customer observation noise covariance matrix and the Kalman gain calculation formula, and the posterior state estimate at time t is updated based on the posterior state update formula and the customer state prior estimate at time t to obtain the Kalman gain matrix K(t) at time t, wherein the Kalman gain calculation formula is: Where K(t) is the Kalman gain matrix at time t; H s is the customer state observation matrix; R s is the client observation noise covariance matrix; The posterior state update formula is: Where, is the posterior state estimation vector at time t; is the prior estimate of the customer status at time t; O s (t) is the customer observation vector at time t; is the predicted observation value based on the prior state estimate.

9. The method for processing multimodal interactive multimedia call center data according to claim 8, characterized in that: In step S6, the step of generating an interactive guidance strategy including risk warning, behavior recommendation, and system intervention instructions based on the current customer state vector, the agent state vector, and the future state prediction result includes: When the estimated value of customer emotional valence <The preset value A of the estimated value of the customer's emotional valence, and the change rate of the customer's emotional valence < preset value B seconds, a high-risk warning is triggered, where: is the estimated value of the customer's emotional valence at time t; is the estimated value of the customer's emotional valence at time (t-1), and Δt is the sampling time interval; Based on solving the constrained programming model, the behavior recommendation at time (t+1) is generated by solving the constrained quadratic programming problem. The formula for solving the constrained programming model is: Where U(t) is the agent behavior vector to be optimized; It is the set of agent operations allowed in the current business scenario; S target is the target customer state vector, which is preset by business rules; is the client state vector at time (t+1) predicted by the adaptive coupled state space model; S target is the target customer state vector; st is the discrete time step; ||·|| is the Euclidean norm; When the agent's cognitive load index >The preset value C of the agent cognitive load index and the customer problem solving progress When the customer problem solving progress reaches the preset value D, the system intervention instruction is triggered and the work order is transferred to the senior support team, where: is the estimated value of the agent's cognitive load index at time t; is the estimated progress of solving customer problems at time t.

10. A multi-modal interactive multimedia call center data processing system, characterized in that: A multimodal interactive multimedia call center data processing method according to any one of claims 1 to 9, comprising: The data processing module is used to collect voice, text, and video interaction data between customers and agents in real time, extract the client's voice acoustic features, text emotion polarity features, and facial action unit features, as well as the agent's operational behavior features, and generate customer observation vectors and agent observation vectors respectively; A model training module, configured to perform offline learning on the parameters of the observation equation and the state transfer equation using historical interaction data to obtain a calibrated adaptive coupled state-space model; An online state estimation module is used to input the customer observation vector and agent observation vector generated in real time by the data processing module into the calibrated adaptive coupled state space model obtained by the model training module, and estimate the current customer state vector and agent state vector online through the extended Kalman filter algorithm, predict the future multi-step state evolution trend, and generate future state prediction results; A strategy generation module is used to generate an interactive guidance strategy including risk warning, behavior recommendation and system intervention instructions based on the current customer state vector, the agent state vector and the future state prediction result generated by the online state estimation module.

Images