Interactive method, device, equipment, storage medium and program product for real-time call scenario

Abstract

The application discloses a kind of real-time call scene interaction method, device, equipment, storage medium and program product, obtain the uplink data stream and downlink data stream of real-time call party;Wherein, the uplink data stream and the downlink data stream include audio signal and video signal;Respectively to the uplink data stream and the downlink data stream are carried out multimodal feature extraction, obtain uplink data feature and downlink data feature;The uplink data feature and the downlink data feature are cooperatively analyzed to identify the current interactive event of the real-time call party;Generate the prompt information corresponding to the interactive event, and display the prompt information in real-time call interface.It is to be used in the present application embodiment, can the audio and video stream of uplink and downlink be carried out cross-modal cooperative analysis, provide targeted interaction optimization suggestion for user, realize experience upgrading and efficiency improvement under real-time call scene.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to an interactive method, apparatus, device, storage medium, and program product for real-time call scenarios. Background Technology

[0002] In real-time communication scenarios, such as video calls, online education, intelligent customer service, and conferencing systems, the analysis of transmitted audio and video streams often involves processing each stream independently. This might involve only analyzing the video feed for facial recognition and gesture recognition, or converting the speech to text using Automatic Speech Recognition (ASR) and then performing separate sentiment analysis, sometimes even relying on pre-defined rules to infer user emotions and satisfaction. This separate processing approach completely ignores the dynamic interaction between voice and video, and fails to capture real-time feedback in two-way interactions. For example, when a user raises their voice, the other party might react with emotions or actions such as frowning or turning their head, leading to a severely distorted understanding of the true emotions and interactive intentions of both parties in the call. Therefore, existing technologies have gaps in bidirectional temporal relationship modeling, and cannot effectively associate the temporal dependencies of uplink and downlink flows. That is, they cannot associate the audio and video sent by the user end with the audio and video sent by the other end, resulting in insufficient capabilities in cross-modal collaborative analysis. It is difficult to integrate multimodal information such as voice emotion, facial expressions, and body movements, and thus cannot accurately identify such key interactive events. This makes it impossible for traditional solutions to provide users with targeted interaction optimization suggestions, which seriously restricts the experience upgrade and efficiency improvement in real-time call scenarios. Summary of the Invention

[0003] The purpose of this invention is to provide an interactive method, device, equipment, storage medium, and program product for real-time call scenarios, which can perform cross-modal collaborative analysis of uplink and downlink audio and video streams, provide users with targeted interaction optimization suggestions, and achieve experience upgrades and efficiency improvements in real-time call scenarios.

[0004] To achieve the above objectives, embodiments of the present invention provide an interactive method for real-time call scenarios, including: The uplink and downlink data streams of both parties in a real-time call are acquired; wherein, both the uplink and downlink data streams include audio signals and video signals. Multimodal feature extraction is performed on the uplink data stream and the downlink data stream respectively to obtain uplink data features and downlink data features; The uplink data features and downlink data features are analyzed collaboratively to identify the current interaction events between the two parties in the real-time call; Generate a prompt message corresponding to the interactive event and display the prompt message in the real-time call interface.

[0005] As an improvement to the above scheme, before performing multimodal feature extraction on the uplink data stream and the downlink data stream respectively, the method further includes: A dynamic timestamp alignment algorithm is used to synchronize the uplink and downlink data streams in time.

[0006] As an improvement to the above scheme, both the uplink data features and the downlink data features include voice features and video features.

[0007] As an improvement to the above scheme, the step of extracting multimodal features from the uplink data stream and the downlink data stream respectively to obtain uplink data features and downlink data features includes: Speech recognition is performed on the speech signals in the uplink data stream and the downlink data stream respectively to obtain speech text content and emotional features, and the speech text content and the emotional features are used as speech features; Visual behavior recognition is performed on the video signals in the uplink data stream and the downlink data stream respectively to obtain visual behavior features, and the visual behavior features are used as video features.

[0008] As an improvement to the above scheme, the visual behavioral features include at least one of facial expression features, head movement features, and body posture features.

[0009] As an improvement to the above scheme, the collaborative analysis of the uplink data features and the downlink data features to identify the current interaction events between the two parties in the real-time call includes: A multi-head cross-attention mechanism is used to perform multimodal alignment and fusion on the uplink data features and the downlink data features to obtain fused multimodal features; The fused multimodal features are input into a bidirectional temporal recurrent neural network to capture the temporal dependencies of the fused multimodal features. The temporal dependencies are input into an attention-based encoding layer to obtain enhanced temporal features; Calculate the correlation score between the uplink data features and the downlink data features in the enhanced time-series features, and use the correlation score to determine the current interaction events between the two parties in the real-time call.

[0010] As an improvement to the above scheme, a multi-head cross-attention mechanism is adopted to perform multimodal alignment and fusion of the uplink data features and the downlink data features to obtain fused multimodal features, including: Based on any feature in the uplink data features and any feature in the downlink data features, a cross-modal key-value pair is formed; The cross-modal key-value pairs are weighted using a multi-head cross-attention mechanism to obtain cross-modal attention weighted features; After adjusting the contribution of the cross-modal attention-weighted features through a gating mechanism, fused multimodal features are obtained.

[0011] As an improvement to the above scheme, the step of determining the current interaction events between the two parties in the real-time call using the correlation score includes: An attention-weighted dual-path joint representation is generated using the aforementioned relevance score formula; The dual-path joint representation is input into a conditional random field model to model the temporal dependency of the dual-path joint representation and output the probability distribution of various interactive events. By combining the preset interactive event determination rules, the probability distribution output by the conditional random field model is analyzed to identify the current interactive events between the two parties in the real-time call.

[0012] As an improvement to the above scheme, the interactive events include at least one of emotional synchronization events, emotional conflict events, and meaningless interactive events; wherein, the emotional synchronization event indicates that the emotional states of the two parties in the real-time call belong to the same-direction correlation mode, the emotional conflict event indicates that the emotional states of the two parties in the real-time call belong to the opposing mode, and the meaningless interactive event indicates that the emotional states of the two parties in the real-time call are unrelated.

[0013] To achieve the above objectives, embodiments of the present invention also provide an interactive device for real-time call scenarios, comprising: The data stream acquisition module is used to acquire the uplink and downlink data streams of both parties in a real-time call; wherein, both the uplink and downlink data streams include audio signals and video signals; A multimodal feature extraction module is used to extract multimodal features from the uplink data stream and the downlink data stream respectively, to obtain uplink data features and downlink data features; An interactive event recognition module is used to perform collaborative analysis on the uplink data features and the downlink data features to identify the current interactive events between the two parties in the real-time call; The prompt message generation module is used to generate prompt messages corresponding to the interactive event; The prompt information display module is used to display the prompt information in the real-time call interface.

[0014] To achieve the above objectives, embodiments of the present invention also provide an interactive device for a real-time call scenario, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the interactive method for a real-time call scenario as described in any of the above embodiments.

[0015] To achieve the above objectives, embodiments of the present invention also provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform an interactive method for a real-time call scenario as described in any of the above embodiments.

[0016] To achieve the above objectives, embodiments of the present invention also provide a computer program product, including computer instructions, which, when executed by a processor, implement the interactive method for a real-time call scenario as described in any of the above embodiments.

[0017] Compared with existing technologies, the interactive methods, devices, equipment, storage media, and program products for real-time call scenarios disclosed in this invention can perform cross-modal collaborative analysis of uplink and downlink audio and video streams, providing users with targeted interaction optimization suggestions and achieving experience upgrades and efficiency improvements in real-time call scenarios. Furthermore, by simultaneously acquiring dual-channel data streams containing audio and video signals from both parties in a real-time call, and accurately identifying interactive events and generating targeted prompts through multimodal feature extraction and collaborative analysis, this not only solves the problem that traditional single-channel audio and video separation processing cannot capture the dynamics of two-way interaction and suffers from distortion in understanding emotions and intentions, but also guides both parties in the call to optimize their communication methods through real-time interface prompts, effectively improving the interaction quality and communication efficiency of real-time calls and adapting to multiple scenario needs. Attached Figure Description

[0018] Figure 1 This is a flowchart of an interactive method for a real-time call scenario provided by an embodiment of the present invention; Figure 2 This is another flowchart of an interactive method for a real-time call scenario provided by an embodiment of the present invention; Figure 3 This is a flowchart of determining the current interaction events between two parties in a real-time call, provided by an embodiment of the present invention; Figure 4 This is a structural block diagram of an interactive device for a real-time call scenario provided in an embodiment of the present invention; Figure 5 This is a structural block diagram of an interactive device for a real-time call scenario provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] See Figure 1 , Figure 1 This is a flowchart illustrating an interactive method for a real-time call scenario provided by an embodiment of the present invention. The interactive method for the real-time call scenario includes steps S1 to S4. See also... Figure 2 , Figure 2 This is another flowchart of an interactive method for a real-time call scenario provided by an embodiment of the present invention, which is described below in conjunction with... Figure 2 Steps S1 to S4 are explained in detail.

[0021] S1. Obtain the uplink and downlink data streams of both parties in a real-time call; wherein, the uplink and downlink data streams both include audio signals and video signals.

[0022] For example, the two parties in a real-time call are a first user (such as the user initiating the call) and a second user (such as the user receiving the call). The uplink data stream includes video signals (including visual information such as the first user's facial expressions and body movements) and audio signals (including auditory information such as the first user's voice content, tone, and emotion) collected and uploaded from the first user's terminal. The downlink data stream includes video signals (including visual feedback such as the second user's facial expressions and body movements) and audio signals (including auditory feedback such as the second user's voice response, tone, and emotion) collected and returned from the second user's terminal. By synchronously acquiring dual data streams, a complete multimodal data foundation is provided for subsequent two-way interactive analysis.

[0023] Furthermore, before performing step S2, the method further includes: using a dynamic timestamp alignment algorithm to perform time synchronization on the uplink data stream and the downlink data stream.

[0024] For example, firstly, the local clocks of both terminals in a real-time call are calibrated using NTP (Network Time Protocol) to eliminate time inconsistencies caused by time deviations in the terminal systems. Then, the PTS (Presentation Time Stamp) of each frame of audio and video in both the uplink and downlink data streams is extracted simultaneously; these are the playback timestamps of the audio and video frames. A dynamic timestamp alignment algorithm is used to calculate the offset of the two streams on the time axis, compensating for frames with delays or advances. For instance, if the PTS of the uplink video frame is t1 and the PTS of the downlink video frame is t2, and the time difference exceeds a preset threshold (e.g., 50 milliseconds), the downlink stream frame is synchronized to time point t1 using a timestamp mapping relationship. This ultimately achieves precise alignment of the uplink and downlink data streams at the frame level (including audio sampling points and video image frames).

[0025] In this embodiment of the invention, the dynamic alignment method combining the NTP protocol and PTS timestamp effectively solves the problem of audio and video timing misalignment caused by network transmission delay and differences in terminal hardware in real-time calls, ensuring the consistency of uplink and downlink in the time dimension. This provides an accurate timing benchmark for subsequent multimodal feature extraction and collaborative interaction event recognition, avoids feature association deviation caused by data misalignment, and improves the analysis accuracy and reliability of the entire method.

[0026] S2. Perform multimodal feature extraction on the uplink data stream and the downlink data stream respectively to obtain uplink data features and downlink data features.

[0027] For example, multimodal feature extraction is used to mine key information reflecting the state and behavior of both parties in a call from audio and video signals. This transforms the raw audio and video data streams into structured features suitable for subsequent analysis. Uplink data features characterize the real-time state of the first user, including speech text content, speech rate, volume, and Valence / Arousal features extracted from the uplink audio signal, as well as facial expressions (such as mouth corner curvature and eyebrow shape changes), head movements (such as nodding and shaking), and upper body posture, etc., extracted from the uplink video signal. Downlink data features characterize the real-time feedback state of the second user, including speech response content, tone, and emotional features extracted from the downlink audio signal, as well as facial expression feedback, gaze direction, and body movements, etc., extracted from the downlink video signal. Multimodal feature extraction lays the data foundation for subsequent collaborative analysis of the two-channel features.

[0028] Furthermore, both the uplink data features and the downlink data features include voice features and video features, and step S2 specifically includes steps S21 to S22.

[0029] S21. Perform speech recognition on the speech signals in the uplink data stream and the downlink data stream respectively to obtain speech text content and emotional features, and use the speech text content and the emotional features as speech features.

[0030] For example, the speech recognition employs a non-autoregressive Paraformer model, combining Voice Activity Detection (VAD) and Speaker Diarization techniques to distinguish overlapping speech. A Conformer-based encoder fuses local temporal features with a global attention mechanism to achieve real-time transcription of mixed Chinese and English speech, yielding the spoken text content. The core of this Conformer-based encoder's real-time transcription of mixed Chinese and English speech lies in the synergistic design of convolutional enhancement and attention mechanisms to accurately capture local details and global context of the speech signal, thereby efficiently handling the linguistic characteristics of mixed Chinese and English scenarios, such as tonal differences, rhythmic variations, and lexical mixing.

[0031] Simultaneously, a deformable speech Transformer is used for sentiment analysis. This deformable speech Transformer is a variant of the Transformer optimized for tasks such as speech emotion recognition. It adaptively adjusts the size and offset of the attention window through a lightweight decision network, eliminating the need for a pre-defined fixed window. It dynamically focuses on key regions containing emotional information based on the characteristics of the input speech. For example, when recognizing strongly emotional speech such as "anger," the model automatically expands the attention window to capture the complete emotional rhythm or adjusts the offset to focus on the emotionally charged speech segment, thus more accurately recognizing multi-granular emotions such as sadness, joy, and anger.

[0032] For example, an adaptive attention window size and offset can be generated using a lightweight decision network. First, the raw predicted values ​​of the window parameters are obtained, satisfying the following formula: (1); in, For the first The time step, the first The window size of each attention head is the original predicted value, without being constrained by the activation function; For the first The time step, the first The original predicted values ​​of the attention head offsets are not constrained by the activation function; For the first The time step, the first The query feature vector of each attention head is obtained by linear transformation of the input speech features and is used to represent the speech feature information of the current time step. For the first The learnable parameter matrix (decision network weights) corresponding to each time step is used to map the query features to the original predicted values ​​of the window parameters. Formula (1) uses a lightweight decision network to initially predict the original values ​​of the attention window size and offset of each attention head at each time step based on the input query features.

[0033] Then, the window size and offset values ​​are constrained to satisfy the following formula: (2); (3); in, For the first The time step, the first The final window size of each attention head is actually used to determine the length of the speech segment for which attention is focused; The first activation function (usually the sigmoid function) is used to apply the original prediction value to the target value. Perform mapping to avoid negative or excessively large window sizes; The second activation function (usually the Tanh function) is used to apply the original prediction value to the target value. Perform mapping, limiting the offset direction and magnitude; The maximum time step of the preset speech sequence (such as the total number of frames in the entire speech segment) is used as the scaling reference for the window size; For the first The time step, the first The final offset of each attention head is used to adjust the center position of the window, shifting it left or right. Formula (2) applies an activation function and range scaling to the original predicted window size to achieve the final window size. ∈(0,L), meaning less than the total length of the speech sequence, ensuring the window is effective. Formula (3) applies an activation function and range scaling to the original predicted offset, making the final offset... ∈(-L,L), meaning the offset range does not exceed the total length of the speech sequence, thus avoiding the window from exceeding the effective speech range.

[0034] After applying window size and offset constraints, based on the current attention head position and offset... Determine key anchor points and window boundaries. , Focusing only on local features within the window, the weighted result is output, satisfying the following formula: (4); in, For the first The weighted result of local features of the j-th attention head at the j-th time step; For the first The key matrix of the j-th time step and the j-th attention head; For the first The matrix of values ​​for the j-th attention head at each time step; For the first The window boundary of the j-th time step and the j-th attention head. The left boundary It has boundaries; To set dimension values; This indicates that only the window in the key matrix is ​​selected. Similarly, calculate the local features within the area. Indicates a window in the value-only matrix The local features within are calculated.

[0035] Finally, multi-head fusion is performed to satisfy the following formula: (5); in, It is an abbreviation for Deformable Self-Attention, which refers to the output of the attention mechanism after being adjusted by a dynamic window; Indicates will The local weighted result of all attention heads at each time step ( arrive (Synthesize along the feature dimension) The output linear transformation matrix is ​​used to map the concatenated multi-head features to output features of a uniform dimension.

[0036] In the above formulas (1) to (4), a lightweight decision network is first used to predict the size and direction of the attention window at each time step. Then, the activation function is used to constrain the window size and offset within a reasonable range to avoid redundant computation caused by an excessively large window or feature loss caused by an excessively small window. Then, attention weights are calculated only for the local features within the window and weighted with the value features to obtain local key information. Finally, by fusing the results from multiple attention heads, we obtain emotional features that include both key local features (such as vocal details at emotional outbursts) and multi-dimensional information. .

[0037] In this embodiment of the invention, existing speech emotion analysis uses a fixed window attention mechanism, which is difficult to capture long-term dependence and sudden emotional changes. This invention dynamically focuses on key speech segments by adaptively adjusting the size and offset of the attention window, which can improve the accuracy of emotion recognition in noisy environments (such as multiple people's overlapping speech).

[0038] S22. Visual behavior recognition is performed on the video signals in the uplink data stream and the downlink data stream respectively to obtain visual behavior features, which are then used as video features. The visual behavior features include at least one of facial expression features, head movement features, and body posture features.

[0039] For example, visual behavior recognition includes at least one of facial expression recognition, head movement recognition, and body posture recognition. Examples of these three recognition methods are given below. It should be noted that the models used in the following recognition methods are only examples. In practical applications, other models with equivalent functions can be used as long as they can complete the recognition.

[0040] 1) Facial expression recognition: The YOLOv8 model can be used to identify facial expressions in video signals. As a high-efficiency object detection model, YOLOv8 provides various pre-trained models of different scales (such as YOLOv8n, YOLOv8s, YOLOv8m, YOLOv8l, and YOLOv8x), with progressively increasing parameter count, computational complexity, and inference speed, suitable for different scenarios requiring a balance between accuracy and efficiency. For real-time video stream processing needs, the YOLOv8s version is selected, as its design achieves an optimal balance between lightweight design and performance.

[0041] It's important to note that during training, OLOv8s uses pre-trained YOLOv8s weights, freezes the backbone parameters, and trains only the head detection and expression classification layers. The pre-trained weights are parameters saved after the model has been trained on a large-scale general image dataset. These parameters have already learned common image features (such as edges, textures, and basic shapes), eliminating the need for training from scratch and serving as a direct starting point for transfer learning. The backbone is the core of YOLOv8s' feature extraction, consisting of multiple convolutional and pooling layers. Its function is to extract low-level features (such as pixel textures) to high-level semantic features (such as facial contours and limb structures) from the input image layer by layer. In subsequent training, the existing parameters of the backbone are not updated (i.e., the general feature extraction capabilities learned during pre-training are fixed). The backbone trained on a general dataset already possesses strong basic feature extraction capabilities; freezing these parameters avoids overfitting due to limited data for new tasks and reduces computational cost (eliminating the need to update a large number of backbone parameters). Training only the detection head and expression classification layer is to focus on the specific needs of the new task. The parameters of these two layers are updated with a small amount of labeled data (such as facial expression samples in a call scene), so that the model can quickly adapt to the visual behavior recognition task of real-time calls.

[0042] 2) Head motion recognition: The 6DRepNet model can be used to extract the 3D position and orientation information of the head from each frame. This algorithm can derive the head's pose parameters for six degrees of freedom, including three translations (x, y, z) and three rotations (rotation angles around the x, y, and z axes). Head movements are typically dynamic, involving a series of pose changes. Therefore, in addition to estimating the head pose for each frame using 6DRepNet, it is necessary to analyze the pose change sequence of consecutive frames and use a Long Short-Term Memory (LSTM) network to capture these dynamic changes and identify specific head movements (nodding, shaking, tilting, etc.).

[0043] 3) Upper body pose recognition can be achieved using GatedUniPose, which integrates UniRepLKNet and gated convolution to enhance robustness in occluded scenarios. Video frames are processed by this model to obtain key points of the upper body. Based on a spatiotemporal graph convolutional network (ST-GCN), a sequence of 30 consecutive key points is modeled as a spatiotemporal graph structure to extract the spatiotemporal features of the action. For example, for a forward leaning motion, the change in the angle between the torso and the horizontal plane needs to be analyzed.

[0044] In this embodiment of the invention, by extracting facial expression features, head movement features, and body posture features from uplink and downlink video signals and integrating them into video features, it is possible to comprehensively capture the non-verbal interaction information of both parties in a real-time call. This process breaks through the limitations of the traditional single-dimensional approach that relies solely on audio signals. It accurately reflects emotional states through facial expressions and supplements interactive intentions through head movements and body postures, providing rich visual dimension basis for subsequent dual-path feature collaborative analysis. This improves the comprehensiveness and accuracy of interactive event recognition and makes the generated prompts more consistent with the actual interactive state of both parties in the call.

[0045] S3. Perform collaborative analysis on the uplink data features and the downlink data features to identify the current interaction events between the two parties in the real-time call.

[0046] For example, the core idea of ​​collaborative analysis is to simultaneously perform temporal modeling on uplink and downlink data streams, analyzing their interactions and feedback. By combining multimodal audio and video data, modeling their temporal dependencies allows for a more comprehensive understanding and optimization of the user's call experience. By constructing a temporal correlation model of dual-path features (such as calculating feature matching degree and sentiment correlation within a time window), the dynamic interaction relationship between the two in the time dimension can be explored. For instance, the correlation calculation between uplink "angry voice + frowning expression" and downlink "silent voice + head turning action" can determine whether causal interaction exists. Interaction events are used to characterize the overall interaction state of both parties in a real-time call within a specific time window. They are an abstract summary of the synergistic effect of dual-path multimodal features, intuitively reflecting the communication quality and interaction patterns of both parties, providing a decision-making basis for generating targeted prompts.

[0047] It should be noted that the analysis goal of the uplink data stream is to reflect the user's emotions and intentions by identifying the user's emotional state, voice content, and facial expressions. The analysis goal of the downlink data stream is to capture the other party's feedback in a timely manner by identifying their reactions (such as voice, facial expressions, head movements, etc.) to understand their emotional changes or behavioral responses.

[0048] Further, see Figure 3 , Figure 3 This is a flowchart of determining the current interaction events between two parties in a real-time call, provided by an embodiment of the present invention. Step S3 specifically includes S31 to S34.

[0049] S31. A multi-head cross-attention mechanism is adopted to perform multimodal alignment and fusion on the uplink data features and the downlink data features to obtain fused multimodal features.

[0050] For example, the multi-head cross-attention mechanism uses uplink data features as queries and downlink data features as keys and values. Multiple parallel attention heads calculate the association weights of the dual-path features in different subspaces, then concatenate and linearly transform the outputs to capture the interaction relationships between uplink and downlink features from multiple angles. For instance, one attention head focuses on the matching degree of voice emotion, while another focuses on the response delay of visual behavior, thus comprehensively exploring the synergistic associations of the dual-path features. Multimodal alignment and fusion aim to eliminate the differences between uplink and downlink data in modal dimensions (speech and video) and temporal dimensions. Through the weight calculation of cross-attention, the uplink voice emotion features, visual behavior features, and corresponding downlink features are accurately associated with each other within a time window and fused into a unified feature vector. This transforms the originally independent uplink and downlink features into holistic features that reflect the two-way interaction logic, providing a foundation for the accurate recognition of subsequent interactive events by incorporating multimodal interaction information.

[0051] Further, step S31 specifically includes: forming a cross-modal key-value pair based on any feature in the uplink data features and any feature in the downlink data features; performing weighted calculation on the cross-modal key-value pair using a multi-head cross-attention mechanism to obtain cross-modal attention-weighted features; and adjusting the contribution of the cross-modal attention-weighted features through a gating mechanism to obtain fused multimodal features.

[0052] For example, firstly, multiple sets of cross-modal key-value pairs are constructed: for instance, uplink speech emotion features (such as Valence and Arousal) are used as queries, and downlink facial expression features (such as smile probability and frown intensity) are used as keys and values; simultaneously, uplink head movement features (such as nodding frequency) are used as queries, and downlink speech emotion features are used as keys, forming multi-dimensional interactive associations. Next, a multi-head cross-attention mechanism is used, with multiple parallel attention heads calculating the relevance weights of different key-value pairs. For example, one attention head focuses on the temporal matching degree between "uplink angry speech" and "downlink frowning expression," while another head calculates the response delay between "uplink questioning speech" and "downlink nodding action." The outputs of each head are concatenated and linearly transformed to obtain cross-modal attention-weighted features, achieving precise alignment of dual-path features in both temporal and modal dimensions. Finally, the contribution of features is dynamically adjusted through a gating mechanism. For example, when the detected speech signal is clear (e.g., no noise, clear semantics), the weight of speech features is increased through gating parameters; when the video behavior feedback is significant (e.g., violent head shaking), the proportion of video features is increased, effectively resolving modal conflicts, and finally outputting a unified feature vector that integrates uplink and downlink audio and video interaction logic.

[0053] Furthermore, the calculation process of the gating parameters in the gating mechanism satisfies the following formula: (6); in, These are the gating parameters; For the Sigmoid function, output gate parameters ∈[0,1], used for weighted fusion of multimodal features; To set the weights, which are learnable parameters; , These are the extracted audio features and video features, respectively.

[0054] In this embodiment of the invention, by constructing cross-modal key-value pairs, the modal barriers of uplink and downlink data features and the limitations of unidirectional analysis are broken. By using a multi-head cross-attention mechanism, the dynamic correlation of dual-path features is captured from multiple dimensions, achieving precise alignment of audio and video features at the temporal and semantic levels. At the same time, by using a gating mechanism to dynamically adjust the contribution of different modal features according to the scenario, the final generated fused multimodal features not only retain the fine-grained information of uplink and downlink features, but also integrate the deep logic of two-way interaction, providing a more comprehensive and accurate feature foundation for subsequent interactive event recognition, and significantly improving the accuracy of judging interactive states such as emotional synchronization and conflict in complex call scenarios.

[0055] S32. The fused multimodal features are input into a bidirectional temporal recurrent neural network to capture the temporal dependencies of the fused multimodal features using the bidirectional temporal recurrent neural network.

[0056] For example, the bidirectional temporal recurrent neural network used is a bidirectional LSTM. Through the parallel operation of two sub-networks, forward LSTM and backward LSTM, the temporal correlations in the fused multimodal features are modeled from different time directions. The forward LSTM processes the feature sequence in chronological order (e.g., t1→t2), focusing on capturing the real-time impact of upstream behavior on downstream. For example, when the fused features show a sudden increase in upstream speech volume at time t1 and a change in facial expression to frowning at time t2, the forward LSTM learns the causal relationship between the two and identifies the temporal dependency of "volume change triggering immediate facial expression feedback". The backward LSTM processes the feature sequence in reverse chronological order (e.g., t2→t1), focusing on analyzing the delayed effect of downstream feedback on upstream. For example, when the fused features show continuous head nodding at time t1 and a gradual slowdown in upstream speech rate at time t2, the backward LSTM mines the delayed relationship between the two and captures the temporal dependency of "nodding feedback leading to subsequent speech rate adjustment". Through bidirectional modeling, the network can comprehensively cover the temporal features of instant response and delayed interaction in real-time calls, transforming the fused multimodal features into sequence features containing complete temporal logic, and providing temporal dimension feature support for the temporal identification of subsequent interactive events.

[0057] S33. Input the temporal dependency into the attention-based encoding layer to obtain enhanced temporal features.

[0058] For example, a Transformer encoding layer is used as an attention-based encoding layer to further capture long-range temporal correlations and enhance the modeling ability for variable-delay interactions based on the temporal dependencies of the bidirectional LSTM output. First, the temporal feature sequence of the bidirectional LSTM output with a time step of T is divided into multiple consecutive segments. Each segment contains features from several adjacent time steps (e.g., every 5 time steps constitute one segment). Within each segment, the association weights of local features are calculated using a self-attention mechanism to focus on capturing interaction details within a short time window, such as the continuous action association of "question-response" in a dialogue segment. Simultaneously, relative positional encoding is introduced. Instead of traditional absolute position encoding, this method uses a new type of encoding that does not rely on the absolute position of features in the sequence. Instead, it models positional relationships by calculating the relative distance between features at different time steps (e.g., the relative distance between time t2 and time t5 is 3). This allows the model to better adapt to the variable delay scenarios of two-way interaction in real-time calls. For example, after the uplink stream speaks, the downlink stream may not give feedback until 1 or 3 seconds later. Finally, by stacking multiple Transformer encoding layers, it preserves local temporal dependencies while capturing long-range correlations across segments, such as the echo between disagreements at the beginning of a call and the subsequent easing of emotions. The enhanced temporal features output more comprehensively cover the details of short-term interactions and long-term interaction trends in real-time calls compared to unidirectional temporal dependencies, providing stronger temporal feature support for accurately identifying continuous and delayed interactive events.

[0059] S34. Calculate the correlation score between the uplink data features and the downlink data features in the enhanced timing features, and use the correlation score to determine the current interaction events between the two parties in the real-time call.

[0060] For example, a dual-stream co-attention module is designed to dynamically calculate the mutual influence weights between upstream and downstream features, using these weights as the correlation scores between the upstream and downstream data features in the enhanced temporal features. For the first... Uplink data characteristics at each time step and the Downlink data characteristics at each time step The correlation score is calculated according to the following formula: (7); in, For relevance score; It is an exponential function used to map input values ​​to non-negative values, while amplifying the differences between features and preparing for subsequent normalization; It is a learnable weight matrix used to perform linear transformations on uplink and downlink features to capture the potential correlation patterns between them; for The transpose of the matrix; Indicates upward characteristics Through weight matrix After transformation, with downlink characteristics Multiplying the transposes of the matrices yields the original correlation value between the two; M is the total number of time steps. =1,2,...,M, the first The time step and the first All time steps belong to M. Formula (7) normalizes the original correlation values ​​of the uplink and downlink features using the softmax function, obtaining their relative importance weights (i.e., correlation scores) in the time dimension. This score reflects the first time step. The uplink characteristics of the first time step for the second time step The higher the score, the more significant the temporal correlation between the two, indicating the degree of influence of the downward features at each time step.

[0061] Furthermore, the interactive events include at least one of emotional synchronization events, emotional conflict events, and meaningless interactive events; wherein, the emotional synchronization event indicates that the emotional states of the two parties in the real-time call belong to the same-direction correlation mode, the emotional conflict event indicates that the emotional states of the two parties in the real-time call belong to the opposing mode, and the meaningless interactive event indicates that the emotional states of the two parties in the real-time call are unrelated.

[0062] For example, if both parties in a call simultaneously shift from neutral to pleasant emotions within a certain time frame (e.g., smiling and raising their tone of voice when discussing a topic), it is considered an emotional synchronization event; if one party displays anger (e.g., questioning loudly) and the other party exhibits evasive behavior (e.g., frequently turning their head or avoiding eye contact), it is considered an emotional conflict event; if the voice and behavioral characteristics of both parties are not clearly related within the time window (e.g., one party describes work content while the other party provides no emotional feedback), it is considered a meaningless interaction event.

[0063] In this embodiment of the invention, by accurately identifying these three types of interactive events, typical interactive scenarios in real-time calls can be fully covered, and the emotional interaction state of both parties can be completely depicted from three dimensions: collaboration, conflict, and lack of connection, laying the foundation for generating differentiated communication prompts in the future.

[0064] Further, in step S34, determining the current interaction events between the two parties in the real-time call using the correlation score specifically includes: generating an attention-weighted dual-path joint representation using the correlation score formula; inputting the dual-path joint representation into a conditional random field model to model the temporal dependency of the dual-path joint representation using the conditional random field model, and outputting the probability distribution of various interaction events; and analyzing the probability distribution output by the conditional random field model in conjunction with preset interaction event judgment rules to identify the current interaction events between the two parties in the real-time call.

[0065] For example, firstly, based on the relevance score formula, dynamic weights are assigned to the uplink and downlink feature sequences. For instance, when the uplink angry speech is correlated with the downlink head-turning action... When the value approaches 1, the weights of both in the dual-path joint representation are significantly increased, forming a joint feature vector focusing on key interactions. This joint representation is then input into a Conditional Random Field (CRF) model. The CRF learns the feature dependencies between adjacent time steps, such as the temporal correlation between the upward questioning voice at time t1 and the downward silence + frowning at time t2, modeling continuous feature patterns of conflict events and outputting probability distributions for events such as emotional synchronization, conflict, and meaningless interactions. For example, the probability of a conflict event is 0.85, and the probability of emotional synchronization is 0.12. Finally, the probability distribution is analyzed using preset rules to determine the interaction event. If the probability of a conflict event is >0.7 (preset threshold, can be set as needed) and satisfies the feature combination of upward angry emotion + downward evasive action, the current interaction event is determined to be a conflict event. If the probability of emotional synchronization is >0.6 (preset threshold, can be set as needed) and the upward and downward emotional features change in the same direction, it is determined to be an emotional synchronization event.

[0066] In this embodiment of the invention, key interactive features are focused on by attention-weighted dual-path joint representation, and the precise modeling of temporal dependencies by CRF model is combined with the solution to the problem that single time step features are difficult to identify continuous interactive events. Furthermore, the interpretability of event judgment is enhanced by preset rules, and the efficient identification of complex interactive events is achieved. This provides accurate event type basis for the subsequent generation of targeted communication prompts, and improves the real-time call assistance system's ability to understand interactive scenarios and the reliability of decision-making.

[0067] Furthermore, this invention employs a sliding window mechanism (window size 5 seconds, step size 1 second) when processing data streams. CUDA (Compute Unified Device Architecture) stream parallelism is used to overlap computation and data transmission. The current window data (e.g., 5 seconds of audio + video) is copied from the CPU (Central Processing Unit) memory to the GPU (Graphics Processing Unit), allowing parallel processing of the previous window's inference task on the GPU. Simultaneously, a model quantization method is provided, such as FP32→INT8. Using TensorRT (NVIDIA's model optimization tool), the weights and activation values ​​(intermediate results from each layer's computation), originally stored as 32-bit floating-point numbers (FP32), are converted to 8-bit integers (INT8). Model quantization reduces memory usage and computational overhead. Combined with layer fusion technology, real-time processing can be achieved on consumer-grade graphics cards. Layer fusion technology merges multiple consecutive computational layers in a neural network (e.g., convolutional layers, batch normalization layers, activation function layers) into a single operational unit, reducing intermediate data storage and transmission in the computation graph.

[0068] Furthermore, this invention, when identifying interactive events, combines an emotional state machine with interaction quality assessment to construct a two-dimensional analysis logic, enabling dynamic analysis and in-depth understanding of interactive events. The emotional state machine defines seven basic emotions—joy, anger, sorrow, fear, surprise, disgust, and neutrality—and a state transition probability matrix. Using the results of dual-stream analysis as input, it updates the emotional states and transition trends of both parties in real time. For example, when "upward angry voice + downward silent voice + head turning" is detected, the state machine determines the risk of escalation of emotional conflict based on preset transition rules (probability weight of anger → disgust). Simultaneously, based on Dynamic Time Warping (DTW), it calculates the synchronization rate of the actions of both parties, quantifying the timeliness of the interactive response. For example, the smaller the time difference between the end of the upward question and the downward nod, the higher the synchronization rate. This synchronization rate is then used to output an interaction quality score of 0-100. Ultimately, the emotional association results from the emotional state machine are integrated with the interaction quality score: if the emotional state is bidirectionally pleasant and the synchronization rate is >80%, the judgment of emotional synchronization events is strengthened; if the emotional state presents an opposing pattern of "anger → avoidance" and the synchronization rate is <30%, it is confirmed as an emotional conflict event; if the emotional states have no obvious association and the synchronization rate is consistently below the threshold (e.g., 50%), it is judged as a meaningless interaction event. This two-dimensional logic captures the emotional core of the interaction through the emotional state machine and quantifies the collaborative efficiency of the interaction through the synchronization rate, making the identification of interaction events more in line with the dynamic characteristics of real-time calls, and providing multi-dimensional basis for generating accurate feedback in the future.

[0069] S4. Generate a prompt message corresponding to the interactive event and display the prompt message in the real-time call interface.

[0070] For example, based on identified interactive events and structured analysis data, multi-dimensional and scenario-based prompts are generated and dynamically presented through a real-time interface. When an emotionally synchronized event is identified and structured data shows that both parties' Valence curves are rising synchronously and the effective feedback rate is >70%, a positive reinforcement prompt is generated, such as displaying "Both parties are communicating happily, and the current topic can be continued" in an AR-enabled pop-up in the corner of the call interface. If an emotional conflict event is identified and a sudden increase in upward Arousal (such as angry voice) and a downward turn-taking delay >3 seconds (avoidance response) are detected, a conflict mitigation prompt is generated, such as displaying "The other party may feel resistant; it is recommended to lower the volume and pause speaking," while simultaneously displaying a comparison curve of both parties' Dominance to help users perceive the imbalance in interaction. For meaningless interactive events, no prompts are required. In addition, the prompts support layered display. Emergency prompts (such as the risk of conflict escalation) are highlighted with a semi-transparent floating box, while general suggestions are embedded in the edge of the interface with lightweight text prompts to avoid interfering with the call experience and help users adjust their communication strategies in real time.

[0071] By employing dual-channel audio and video stream collaborative analysis and temporal dynamic modeling, this invention achieves accurate emotional understanding and interactive feedback in real-time call scenarios. It can be applied to situations in remote education where ordinary video calls suffer from low interaction efficiency due to teachers' inability to perceive students' comprehension in real time and one-way explanations. This invention enables: ① Synchronous emotional detection: During teacher explanation, the system detects students' nodding actions (video) and voice responses of "understood" (audio), determining a high-match interaction (score 95), prompting the teacher to continue; ② Conflict warning: If a student answers "no problem" but frequently shakes their head, a low-match alarm is triggered, suggesting the teacher repeat the key points.

[0072] Furthermore, in step S4, generating the prompt information corresponding to the interactive event includes: calling a preset mapping library to determine the prompt strategy corresponding to the interactive event; and generating the prompt information according to the prompt strategy.

[0073] For example, the preset mapping library stores the correspondence between interactive events and prompt strategies, covering rules such as event type, structured data threshold, prompt direction, and presentation format. For instance, when an emotional conflict event is identified, calling the mapping library can match the corresponding prompt strategy: with conflict mitigation as the core, it prioritizes prompting the initiator to reduce emotional intensity, and the auxiliary information includes a comparison of the emotional curves of both parties; based on this strategy, specific prompt information is generated, such as "You are currently quite emotional, and the other party has shown a delayed response. It is recommended to slow down your speech first," and displayed in a prominent orange-background prompt box on the left side of the interface.

[0074] In this embodiment of the invention, the process achieves precise binding between interactive events and prompt strategies through a preset mapping library, avoiding the randomness and subjectivity of prompt information. Simultaneously, the prompt information generated based on the strategy closely adheres to the core of the event (e.g., focusing on emotional relief during conflict, and focusing on process optimization during inefficiency), making the prompts more targeted and actionable. Furthermore, the unified strategy framework ensures consistency of prompt logic across different scenarios, enabling real-time intervention to help users adjust their communication methods promptly, and enhancing user perception of the interaction status through data-driven presentation, ultimately effectively improving the quality and efficiency of real-time calls.

[0075] Compared to existing technologies, the interactive method for real-time calls disclosed in this invention can perform cross-modal collaborative analysis of uplink and downlink audio and video streams, providing users with targeted interaction optimization suggestions and achieving experience upgrades and efficiency improvements in real-time call scenarios. Furthermore, by simultaneously acquiring dual-channel data streams containing audio and video signals from both parties in a real-time call, and accurately identifying interactive events and generating targeted prompts through multimodal feature extraction and collaborative analysis, it not only solves the problem that traditional single-channel audio and video separation processing cannot capture the dynamics of two-way interaction and suffers from distortion in understanding emotions and intentions, but also guides both parties in the call to optimize their communication methods through real-time interface prompts, effectively improving the interaction quality and communication efficiency of real-time calls and adapting to multiple scenario needs.

[0076] See Figure 4 , Figure 4 This is a structural block diagram of an interactive device 100 for a real-time call scenario provided in an embodiment of the present invention. The interactive device 100 for a real-time call scenario includes: The data stream acquisition module 11 is used to acquire the uplink data stream and downlink data stream of both parties in a real-time call; wherein, the uplink data stream and the downlink data stream both include audio signals and video signals; The multimodal feature extraction module 12 is used to extract multimodal features from the uplink data stream and the downlink data stream respectively, to obtain uplink data features and downlink data features; The interactive event recognition module 13 is used to perform collaborative analysis on the uplink data features and the downlink data features to identify the current interactive events between the two parties in the real-time call; The prompt message generation module 14 is used to generate prompt messages corresponding to the interactive event; The prompt information display module 15 is used to display the prompt information in the real-time call interface.

[0077] Specifically, the interactive device 100 for the real-time call scenario further includes: The timing synchronization module is used to perform timing synchronization of the uplink data stream and the downlink data stream using a dynamic timestamp alignment algorithm.

[0078] It is worth noting that the working process of each module in the interactive device 100 for the real-time call scenario described in the embodiments of the present invention can refer to the working process of the interactive method for the real-time call scenario described in the above embodiments, and will not be repeated here.

[0079] See Figure 5 , Figure 5 This is a structural block diagram of an interactive device 200 for real-time call scenarios provided in an embodiment of the present invention. The interactive device 200 for real-time call scenarios includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, it implements the steps in the interactive method embodiments of the various real-time call scenarios described above.

[0080] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the interactive device 200 of the real-time call scenario.

[0081] The interactive device 200 for the real-time call scenario may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of the interactive device 200 for the real-time call scenario and does not constitute a limitation on the interactive device 200 for the real-time call scenario. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the interactive device 200 for the real-time call scenario may also include input / output devices, network access devices, buses, etc.

[0082] The processor 21 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 21 is the control center of the interactive device 200 in the real-time call scenario, connecting various parts of the interactive device 200 throughout the real-time call scenario using various interfaces and lines.

[0083] The memory 22 can be used to store the computer programs and / or modules. The processor 21 implements various functions of the interactive device 200 in the real-time call scenario by running or executing the computer programs and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0084] In this context, the modules / units integrated into the interactive device 200 for the real-time call scenario, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 21, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0085] Furthermore, the present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements an interactive method for a real-time call scenario as described in any of the above embodiments.

[0086] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. An interactive method for real-time call scenarios, characterized in that, include: The uplink and downlink data streams of both parties in a real-time call are acquired; wherein, both the uplink and downlink data streams include audio signals and video signals. Multimodal feature extraction is performed on the uplink data stream and the downlink data stream respectively to obtain uplink data features and downlink data features; The uplink data features and downlink data features are analyzed collaboratively to identify the current interaction events between the two parties in the real-time call; Generate a prompt message corresponding to the interactive event and display the prompt message in the real-time call interface; The uplink data features and downlink data features both include voice features and video features; the collaborative analysis of the uplink data features and downlink data features to identify the current interaction events between the two parties in the real-time call includes: A multi-head cross-attention mechanism is used to perform multimodal alignment and fusion on the uplink data features and the downlink data features to obtain fused multimodal features; The fused multimodal features are input into a bidirectional temporal recurrent neural network to capture the temporal dependencies of the fused multimodal features. The temporal dependencies are input into an attention-based encoding layer to obtain enhanced temporal features; Calculate the correlation score between the uplink data features and the downlink data features in the enhanced time-series features, and use the correlation score to determine the current interaction events between the two parties in the real-time call.

2. The interactive method for a real-time call scenario as described in claim 1, characterized in that, Before performing multimodal feature extraction on the uplink data stream and the downlink data stream respectively, the method further includes: The uplink and downlink data streams are synchronized using a combination of NTP and PTS.

3. The interactive method for a real-time call scenario as described in claim 1, characterized in that, The step of extracting multimodal features from the uplink data stream and the downlink data stream respectively to obtain uplink data features and downlink data features includes: Speech recognition is performed on the speech signals in the uplink data stream and the downlink data stream respectively to obtain speech text content and emotional features, and the speech text content and the emotional features are used as speech features; Visual behavior recognition is performed on the video signals in the uplink data stream and the downlink data stream respectively to obtain visual behavior features, and the visual behavior features are used as video features.

4. The interactive method for a real-time call scenario as described in claim 3, characterized in that, The visual behavioral features include at least one of facial expression features, head movement features, and body posture features.

5. The interactive method for a real-time call scenario as described in claim 1, characterized in that, The multi-head cross-attention mechanism is used to perform multimodal alignment and fusion of the uplink data features and the downlink data features to obtain fused multimodal features, including: Based on any feature in the uplink data features and any feature in the downlink data features, a cross-modal key-value pair is formed; The cross-modal key-value pairs are weighted using a multi-head cross-attention mechanism to obtain cross-modal attention weighted features; After adjusting the contribution of the cross-modal attention-weighted features through a gating mechanism, fused multimodal features are obtained.

6. The interactive method for a real-time call scenario as described in claim 1, characterized in that, The step of determining the current interaction events between the two parties in the real-time call using the correlation score includes: An attention-weighted dual-path joint representation is generated using the aforementioned relevance scoring formula; The dual-path joint representation is input into a conditional random field model to model the temporal dependency of the dual-path joint representation and output the probability distribution of various interactive events. By combining the preset interactive event determination rules, the probability distribution output by the conditional random field model is analyzed to identify the current interactive events between the two parties in the real-time call.

7. The interactive method for a real-time call scenario as described in claim 6, characterized in that, The interactive events include at least one of emotional synchronization events, emotional conflict events, and meaningless interactive events; wherein, the emotional synchronization event indicates that the emotional states of the two parties in the real-time call belong to the same-direction correlation mode, the emotional conflict event indicates that the emotional states of the two parties in the real-time call belong to the opposing mode, and the meaningless interactive event indicates that the emotional states of the two parties in the real-time call are unrelated.

8. An interactive device for real-time call scenarios, characterized in that, include: The data stream acquisition module is used to acquire the uplink and downlink data streams of both parties in a real-time call; wherein, both the uplink and downlink data streams include audio signals and video signals; A multimodal feature extraction module is used to extract multimodal features from the uplink data stream and the downlink data stream respectively, to obtain uplink data features and downlink data features; An interactive event recognition module is used to perform collaborative analysis on the uplink data features and the downlink data features to identify the current interactive events between the two parties in the real-time call; The prompt message generation module is used to generate prompt messages corresponding to the interactive event; A notification display module is used to display the notification information in the real-time call interface; The uplink data features and downlink data features both include voice features and video features; the interactive event recognition module is specifically used for: A multi-head cross-attention mechanism is used to perform multimodal alignment and fusion on the uplink data features and the downlink data features to obtain fused multimodal features; The fused multimodal features are input into a bidirectional temporal recurrent neural network to capture the temporal dependencies of the fused multimodal features. The temporal dependencies are input into an attention-based encoding layer to obtain enhanced temporal features; Calculate the correlation score between the uplink data features and the downlink data features in the enhanced time-series features, and use the correlation score to determine the current interaction events between the two parties in the real-time call.

9. An interactive device for real-time call scenarios, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement an interactive method for a real-time call scenario as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform an interactive method for a real-time call scenario as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement an interactive method for a real-time call scenario as described in any one of claims 1 to 7.

Images