A live broadcast trustworthiness evaluation method and system based on multi-modal time sequence emotion fusion

Abstract

The application discloses a kind of multi-modal time sequence sentiment fusion live broadcast trust degree evaluation method and system, belong to electronic commerce and artificial intelligence technical field, for solving the modal of trust degree evaluation under existing e-commerce live broadcast scene is relatively single, lack of comprehensive use to multi-modal information such as voice, vision, text, and cannot capture the dynamic evolution of trust degree with live broadcast process technical problem.The method comprises: the video stream, audio stream and barrage text stream collected in live room are carried out multi-source live broadcast data time sequence alignment and preprocessing;Multi-modal feature data are extracted and processed about time sequence emotional characteristics;Multi-modal emotional characteristics are processed by cross-modal time sequence fusion;Fusion emotional characteristic sequence is processed based on global context representation mapping of trust degree, to obtain the real-time trust degree score of live room in current period;Real-time trust degree score is carried out risk early warning and dynamic evaluation, to obtain final evaluation result.

Description

Technical Field

[0001] This application relates to the fields of e-commerce and artificial intelligence technology, and in particular to a live streaming trust assessment method and system based on multimodal temporal emotion fusion. Background Technology

[0002] With the rapid development of live-streaming e-commerce, consumer trust has become a core variable determining transaction conversion. However, frequent issues such as false advertising and misleading content by live-streamers have led to a crisis of consumer trust. The trust mechanism in live-streaming e-commerce is shifting from moral constraints to the construction of institutional infrastructure, with privacy and security becoming strategic infrastructure, driving digital trust to become a core competitive advantage in the industry.

[0003] However, existing methods for assessing trust in e-commerce live streaming are relatively singular in terms of modality, lacking the comprehensive application of multimodal information such as voice, vision, and text; they are difficult to capture the temporal dynamic evolution of trust during the live stream; they are not conducive to using general emotion recognition technology to establish an emotion-trust mapping model for the e-commerce live streaming field; and existing trust mechanism research remains at the theoretical level and has not been transformed into an engineering-deployable system.

[0004] Therefore, there is an urgent need for a trust assessment method for e-commerce live streaming based on multimodal temporal sentiment fusion to solve the problem of dynamic quantitative assessment of consumer trust by multimodal temporal sentiment features in e-commerce live streaming. Summary of the Invention

[0005] This application provides a method and system for evaluating trust in live streaming based on multimodal temporal emotion fusion, which addresses the following technical problems: existing trust evaluation modalities in e-commerce live streaming scenarios are relatively singular, lacking the comprehensive application of multimodal information such as voice, vision, and text, and failing to capture the dynamic evolution of trust as the live streaming progresses.

[0006] The embodiments of this application adopt the following technical solutions: On one hand, this application provides a method for evaluating trust in live streaming based on multimodal temporal sentiment fusion, comprising: performing temporal alignment and preprocessing on video streams, audio streams, and bullet screen text streams collected in the live streaming room to obtain multimodal feature data; extracting relevant temporal sentiment features from the multimodal feature data using a preset temporal encoder to obtain multimodal sentiment features; wherein the multimodal sentiment features include: video modal sentiment features, audio modal sentiment features, and text modal sentiment features; performing cross-modal temporal fusion processing on each modal sentiment feature based on multi-head cross-attention using a shared semantic space to obtain a fused sentiment feature sequence; performing global context representation mapping processing based on trust level on the fused sentiment feature sequence to obtain a real-time trust score for the live streaming room in the current period; and performing risk warning and dynamic evaluation on the real-time trust score to obtain a final evaluation result.

[0007] This application's embodiments extract deep emotional features from the anchor's facial micro-expressions, vocal rhythm fluctuations, and real-time consumer feedback text. This allows trust assessment to be based on the mutual corroboration of multi-source information, avoiding the risk of misjudgment when single-modal signals are ambiguous in emotional expression or when data is sparse. Furthermore, the trust score is output in a fixed-period rolling fashion, ensuring that the assessment results are synchronized with the live broadcast process in real time. The dynamic tracking capability allows the platform to receive early warning signals when a decline in trust occurs. It can also directly map temporally fused emotional features into continuous trust scores, enabling quantitative assessment of emotional signals. Simultaneously, a multi-head cross-attention technique is introduced, allowing each modality's features to interact in parallel pairs within a shared semantic space. This facilitates the dynamic capture of complementary cues between modalities and improves the trust prediction rate.

[0008] In one feasible implementation, the video stream, audio stream, and bullet screen text stream collected in the live broadcast room are subjected to time-series alignment and preprocessing of multi-source live broadcast data to obtain multimodal feature data. Specifically, this includes: pulling the video stream, audio stream, and bullet screen text stream in the current live broadcast window in parallel through the live broadcast platform interface to obtain the multi-source live broadcast data; dividing the audio signal in the audio stream into short segments corresponding to the duration of the video frames in the video stream according to the time anchor points of the video frames, and aggregating the text in the bullet screen text stream into a segment sequence according to a fixed time window to obtain the data time-series alignment result of the multi-source live broadcast data; obtaining a time-series aligned multimodal data frame group based on the data time-series alignment result; extracting the data features of different modal data in the multimodal data frame group to obtain the multimodal feature data; wherein, the multimodal feature data includes the anchor's facial image features, speech acoustic signal features, and consumer text feedback features at the same time.

[0009] In one feasible implementation, a preset temporal encoder is used to extract temporal emotional features from the multimodal feature data to obtain multimodal emotional features. Specifically, this includes: using a temporal encoder equipped with a three-dimensional spatiotemporal convolutional network to perform joint convolution processing on the video modal features in the multimodal feature data for facial change features and head pose shift features, and outputting the video modal emotional features reflecting the anchor's real-time emotional state based on single-frame spatial features and motion trends between adjacent frames; and using a temporal encoder equipped with a temporal recursive network to perform prosodic processing on the audio modal features in the multimodal feature data for sound rhythm changes. The system employs attention-focused control of multimodal features and, based on the temporal sequence dependency of a temporal recursive network, automatically focuses on the local audio segments with the strongest emotions in the audio modality features, compresses and outputs the audio modality emotional features. Through a pre-trained language model, it performs contextual semantic representation recognition processing on the text modality features in the multimodal feature data, and inputs the recognized contextual semantic representation into a temporal recursive network to capture the emotional evolution context features between multiple bullet screen texts. The emotional evolution context features are dynamically weighted with preset emotional word intensities to output text modality emotional features with high emotional sensitivity in consumer feedback.

[0010] In one feasible implementation, by sharing a semantic space, cross-modal temporal fusion processing based on multi-head cross-attention is performed on each modal sentiment feature in the multimodal sentiment features to obtain a fused sentiment feature sequence. Specifically, this includes: mapping each modal sentiment feature in the multimodal sentiment features to the shared semantic space of the same dimension using a feedforward projection network; combining the video modal sentiment features, the audio modal sentiment features, and the text modal sentiment features at the same time into modal triples and inputting them into the multi-head cross-attention module; and then... The multi-head cross-attention module performs queries, keys, and values ​​on each modal feature in the modal triplet, and calculates the pairwise association weights within and between each modality in parallel to capture complementary and conflicting information across modalities globally. A pre-defined gated fusion network normalizes the three modal features in the modal triplet after cross-modal interaction to obtain normalized contribution coefficients. Based on these normalized contribution coefficients, the video modal sentiment features, the audio modal sentiment features, and the text modal sentiment features are weighted and synthesized to output a fused sentiment feature sequence that evolves over time.

[0011] In one feasible implementation, the fused emotional feature sequence is subjected to trust-based global context representation mapping processing to obtain the real-time trust score of the live stream in the current period. Specifically, this includes: inputting the fused emotional feature sequence into a preset deep mapping network; wherein the deep mapping network has time-series awareness capabilities; projecting the fused emotional features in the fused emotional feature sequence from the semantic space to the trust evaluation space through multi-layer nonlinear transformations in the deep mapping network; performing quantitative evaluation processing on each time node in the fused emotional feature sequence in the trust evaluation space based on expert rules through the temporal attention mechanism introduced by the deep mapping network, and focusing the key moments affecting trust as global context representation vectors; wherein the key moments include at least: moments when the streamer's emotions are intense and moments when questioning words appear in concentrated bullet comments; mapping the global context representation vector to a continuous scalar value through the regression layer of the deep mapping network, and constraining it to a preset interval through a nonlinear activation function to obtain the real-time trust score of the live stream in the current period; wherein the real-time trust score reflects the consumer's overall trust level towards the streamer and the live stream at the current moment.

[0012] In one feasible implementation, before performing risk warning and dynamic assessment on the real-time trust score to obtain the final assessment result, the method further includes: performing adaptive smoothing filtering on the real-time trust score; dynamically adjusting the weight ratio between the historical score accumulation value and the current score observation value according to the instantaneous fluctuation amplitude of the real-time trust score to obtain the smoothed real-time trust score.

[0013] In one feasible implementation, the real-time trust score is subjected to risk warning and dynamic assessment to obtain a final assessment result. Specifically, this includes: obtaining several real-time trust scores within a fixed time window and determining them as a real-time trust score sequence; if the real-time trust score sequence is continuously increasing, the final assessment result corresponding to the fixed time window period is determined to be in a normal monitoring state; if the real-time trust score sequence is continuously decreasing and within a controllable decreasing range, the final assessment result corresponding to the fixed time window period is determined to be in a state requiring attention; if the real-time trust score sequence is continuously decreasing and within a short-term excessively large drop range, the final assessment result corresponding to the fixed time window period is determined to be in a state requiring mild intervention; if the real-time trust score sequence is continuously decreasing and within a long-term excessively large drop range, the final assessment result corresponding to the fixed time window period is determined to be in a state requiring mandatory intervention.

[0014] In one feasible implementation, after obtaining the final evaluation result, the method further includes: encapsulating and outputting the final evaluation result as structured data to obtain standardized evaluation structure data; wherein, the evaluation structure data includes: live room identification status, timestamp, trust score, confidence interval, warning level, and contribution weight of each modality.

[0015] In one feasible implementation, the shared semantic space includes a video projection layer, an audio projection layer, and a text projection layer; each projection layer consists of two fully connected layers, a residual connection layer, and a layer normalization layer.

[0016] On the other hand, this application also provides a live streaming trust assessment system based on multimodal temporal emotion fusion. The live streaming trust assessment system has instructions that can be executed by a processor so that the processor can execute a live streaming trust assessment method based on multimodal temporal emotion fusion as described in any of the above embodiments.

[0017] This application provides a method and system for evaluating trust in live streaming based on multimodal temporal sentiment fusion. Compared with the prior art, the embodiments of this application have the following beneficial technical effects: 1. By constructing a dedicated temporal encoder, we extract deep emotional features from the anchor's facial micro-expressions, voice rhythm fluctuations, and real-time feedback text from consumers. This allows trust assessment to be based on the mutual corroboration of multi-source information, avoiding the risk of misjudgment when single-modal signals are ambiguous in emotional expression or when data is sparse.

[0018] 2. By continuously pulling live stream data, the trust score is output in a fixed period, so that the evaluation results are synchronized with the live stream process in real time. Moreover, the dynamic tracking capability enables the platform to obtain early warning signals when the trust score shows a downward trend.

[0019] 3. By constructing an end-to-end deep regression network, temporal fusion sentiment features are directly mapped to continuous trust scores, enabling the sentiment signal to be quantitatively evaluated.

[0020] 4. Multi-head cross-attention technology is introduced, in which features of each modality interact in parallel pairs in a shared semantic space, which can dynamically capture complementary cues between modalities and is conducive to improving the trust prediction rate. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 A flowchart of a live streaming trust assessment method based on multimodal temporal sentiment fusion provided in this application embodiment; Figure 2 This is a schematic diagram of the structure of a live streaming trust assessment system based on multimodal temporal emotion fusion, provided in an embodiment of this application. Detailed Implementation

[0022] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0023] This application provides a method for evaluating trust in live streaming based on multimodal temporal sentiment fusion, such as... Figure 1 As shown, the live streaming trust assessment method based on multimodal temporal sentiment fusion specifically includes steps S101-S105: S101. Perform time-series alignment and preprocessing on the video stream, audio stream, and bullet screen text stream collected in the live broadcast room to obtain multimodal feature data.

[0024] Specifically, it is necessary to first retrieve the video stream, audio stream, and bullet screen text stream of the current live stream window in parallel through the live streaming platform interface to obtain multi-source live streaming data.

[0025] In one embodiment, the system pulls video streams (including the anchor's facial image), audio streams (including voice signals), and bullet screen text streams in parallel through the live streaming platform interface. Since the physical generation rates (frame rate, sampling rate, and text arrival rate) of the three data streams differ, directly mixing them would result in timing misalignment, leading to distortion of emotional semantics. Therefore, this solution first uses a unified server clock as a reference to add a millisecond-level timestamp to each video frame, each audio packet, and each bullet screen comment.

[0026] Furthermore, based on the time anchor points of the video frames, the audio signals in the audio stream are divided into short segments corresponding to the duration of the video frames in the video stream, and the text in the bullet screen text stream is aggregated into a segment sequence according to a fixed time window, resulting in the data time-series alignment result of the multi-source live streaming data. Then, based on the data time-series alignment result, the time-series aligned multimodal data frame group can be obtained.

[0027] In one embodiment, an alignment strategy using video frames as time anchors is employed to segment the audio signal along the timeline into short segments corresponding to the duration of the video frames. The bullet comments are then aggregated into a sequence of segments within fixed time windows (utilizing only historical windows and the current moment to avoid introducing future information). Through this processing, the raw, unordered multi-source live streaming data is transformed into a series of strictly time-corresponding multimodal data frame groups. Each group contains the anchor's facial image, acoustic speech signals, and consumer text feedback at the same moment, laying the data foundation for subsequent joint feature extraction.

[0028] Furthermore, it is necessary to extract the data features of different modalities in the multimodal data frame group to obtain multimodal feature data. This multimodal feature data includes the anchor's facial image features, speech acoustic signal features, and consumer text feedback features at the same time point.

[0029] In one embodiment, video acquisition parameters are: sampling frequency fps = 15 frames / second, and resolution uniformly scaled to 224×224 pixels. Face detection and AU feature extraction (using the complete OpenFace workflow): Face detection, keypoint localization, and AU intensity extraction are performed uniformly using the OpenFace 3.0 toolkit. The Constrained Local Model (CLM) built into OpenFace is trained on the IMMFace dataset and can stably output the intensity values ​​of 49 facial keypoints and 17 action unit (AU) points. MediaPipe is not used in conjunction with OpenFace to avoid feature incompatibility issues.

[0030] In one embodiment, the audio acquisition parameters are: sampling rate 16kHz, quantization bit depth 16bit, mono. Pre-emphasis: first-order high-pass filter, coefficient a=0.97. Framing and windowing: frame length 25ms (400 sampling points), frame shift 10ms (160 sampling points), Hamming window. Acoustic feature extraction: using the OpenSMILE toolkit, configuration file MFCC12_0_D_A.conf, extracting 39-dimensional MFCC (13th order + Delta + Delta-Delta). Fundamental frequency F0: autocorrelation method, search range 50-500Hz. Short-time frame energy: E = Σ[x(n)]². Formants F1-F3: linear predictive coding (LPC), order 12. Final output: 44-dimensional features / frame (39+1+1+3). Feature normalization: mean-variance normalization within a sliding window (5 seconds).

[0031] In one embodiment, bullet comments are captured in real time: a WebSocket connection is made to the bullet comment server, and each bullet comment contains text content, user ID (SHA-256 hash anonymized), and timestamp. The bullet comment aggregation strategy (causal alignment) uses a trust assessment period (1 second) as a baseline, aggregating all bullet comments within the previous 5 seconds into a single paragraph every 1 second. Symmetrical windows are not used to avoid using future information, meeting the causal requirements of a real-time system. Cleaning and word segmentation: HTML tags, excessively short bullet comments (<2 characters), and advertising bullet comments are removed (using regular expression matching); pkuseg word segmentation is used, and a custom e-commerce thesaurus is loaded. Semantic encoding: a RoBERTa-wwm-ext pre-trained model is used, with 12 Transformer layers, 768-dimensional hidden layers, and 12 attention heads. The maximum input length is 512, and the [CLS] vector is used as the paragraph semantic vector (768-dimensional). Inference acceleration: on an NVIDIA T4 GPU, using the ONNX Runtime, the processing latency for a single bullet comment is approximately 15ms (approximately 48ms for native PyTorch), meeting real-time requirements.

[0032] S102. Using a preset temporal encoder, extract relevant temporal sentiment features from the multimodal feature data to obtain multimodal sentiment features. These multimodal sentiment features include: video modal sentiment features, audio modal sentiment features, and text modal sentiment features.

[0033] It should be noted that after obtaining the aligned multimodal feature data, a specialized temporal encoder needs to be designed for the signal structure characteristics of each modality feature to extract deep-level sentiment representations. This facilitates the conversion of the original signal into a high-dimensional semantic vector for subsequent cross-modal understanding.

[0034] Specifically, a temporal encoder with a three-dimensional spatiotemporal convolutional network is introduced to perform joint convolution processing on the video modal features in the multimodal feature data, involving facial change features and head pose shift features. Based on the spatial features of a single frame and the motion trend between adjacent frames, the video modal emotional features reflecting the anchor's real-time emotional state are output.

[0035] In one embodiment, for a continuous sequence of video frames, a 3D spatiotemporal convolutional network is used to capture subtle changes in the anchor's facial muscle movements (such as relaxed eyebrows and downturned corners of the mouth) and head posture shifts. The encoder models both the spatial features of a single frame and the motion trends between adjacent frames through joint convolution in the spatiotemporal dimensions, outputting a dynamic visual feature vector that reflects the anchor's real-time emotional state, i.e., outputting video modal emotional features that reflect the anchor's real-time emotional state.

[0036] In one embodiment, a video feature encoder can also be used. Optional architectures: I3D and SlowFast are provided, selected based on computational resources. I3D architecture (default): Based on the Inception-V1 backbone, the convolutional kernels are "inflated" to 3D. Input: 16 consecutive video frames (approximately 1.07 seconds), with a stride of 8 frames (0.53 seconds) for sliding extraction, and adjacent segments overlapping by 50%. To reduce computational redundancy, a feature map caching mechanism is used: intermediate results from convolutions in adjacent windows are reused, with additional computation only occurring within the stride frames. Output: 2048-dimensional spatiotemporal feature vector. SlowFast architecture (high performance): Dual-path: Slow path (16-step time, 1-second frame interval), Fast path (2-step time, 133ms frame interval). Lateral connections fuse features. Output: 2304-dimensional feature vector. Positional encoding: Learnable positional encoding is added to the input layer, with the same dimension as the input features.

[0037] Furthermore, a temporal encoder with a temporal recursive network is introduced to perform attention-focusing control on the audio modal features in the multimodal feature data regarding the prosodic variation features. Based on the temporal sequence dependency of the temporal recursive network, the local audio segments with the most intense emotions in the audio modal features are automatically focused out, compressed, and output as audio modal emotional features.

[0038] In one embodiment, a temporal recurrent network combined with an attention pooling mechanism is used for audio segments. Since emotional fluctuations in speech are often carried by prosodic changes (such as sudden pitch increases or faster speech rates), the recurrent network can transmit these sequence dependencies along the time axis, while the attention mechanism can automatically focus on the local segments in a speech segment with the most intense emotions, compressing and generating concise auditory emotional features, i.e., audio modal emotional features.

[0039] In one embodiment, an audio feature encoder can also be used. Network: Bi-LSTM + Temporal Attention. Hidden layer dimension 256 (bidirectional → 512), number of layers 2, dropout=0.3. Input alignment: Each video segment (16 frames, 1.07 seconds) corresponds to 107 audio frames T_audio (1.07s / 0.01s). For batch processing, zero-padding is used to a fixed length of 128. Temporal attention calculates weights α_t = softmax(W_att·tanh(V_att·h_t + b_att)), and the weighted sum yields a 512-dimensional audio sentiment feature.

[0040] Furthermore, a pre-trained language model is introduced to identify and process the contextual semantic representations of text modal features in the multimodal feature data. The identified contextual semantic representations are then input into a temporal recurrent network to capture the emotional evolution context features between multiple bullet screen texts. Finally, the emotional evolution context features are dynamically weighted with preset sentiment word intensities to output the text modal sentiment features with high emotional sensitivity in consumer feedback.

[0041] In one embodiment, for the aggregated sequence of bullet comments, a domain-adaptive pre-trained language model is first used to obtain the contextual semantic representation of each bullet comment, and then this representation is input into a lightweight temporal recurrent network. This design aims to capture the emotional evolution between multiple bullet comments (such as from teasing to questioning) and introduces dynamic weighting of sentiment word intensity to make it more sensitive to emotional changes in consumer feedback, thereby outputting text modal sentiment features with high emotional sensitivity in consumer feedback.

[0042] In one embodiment, a text feature encoder can also be used. The bullet screen paragraphs slide in 1-second increments, and every second, bullet screens from the previous 5 seconds are retrieved to form a paragraph sequence. The paragraph at the current time t is P_t, and the historical paragraph sequence is [P_{t-4}, P_{t-3}, ..., P_t] (a total of 5 paragraphs, covering 25 seconds of history). Each paragraph is RoBERTa encoded as a 768-dimensional vector. The paragraph sequence is input into a gated recurrent unit (GRU): a single-layer, bidirectional, 256-dimensional hidden layer. The GRU outputs 256-dimensional text temporal features. Sentiment intensity weighting: Before inputting into the GRU, the intensity of sentiment words in the paragraph (sentiment lexicon + number of exclamation marks) is multiplied by a coefficient α_t∈[0,1].

[0043] As a feasible implementation method, after the above-mentioned extraction and processing of temporal emotional features, the video, audio and text are respectively refined into three temporal feature vector sequences with unified dimensions but different physical meanings, so as to jointly depict the real-time emotional mirror between the anchor's expression and the consumer's reaction.

[0044] S103. By sharing the semantic space, cross-modal temporal fusion processing based on multi-head cross-attention is performed on each modal sentiment feature in the multimodal sentiment features to obtain the fused sentiment feature sequence.

[0045] Specifically, a feedforward projection network is needed to map each modal sentiment feature in the multimodal sentiment features to a shared semantic space of the same dimension. The shared semantic space includes a video projection layer, an audio projection layer, and a text projection layer; and each projection layer consists of two fully connected layers, a residual connection, and a layer normalization layer.

[0046] As a feasible implementation method, heterogeneous features between different modalities have natural gaps in numerical distribution and semantic hierarchy. Simple concatenation or weighted averaging will drown out complementary cues between modalities. Therefore, it is necessary to use independent feedforward projection networks to map each modal sentiment feature (three heterogeneous features) in the above-mentioned multimodal sentiment features to a shared semantic space of the same dimension, so as to eliminate the difference in scale and make the three comparable.

[0047] Furthermore, the video modal sentiment features, audio modal sentiment features, and text modal sentiment features at the same time are combined into modal triples and input into a multi-head cross-attention module. Then, through the multi-head cross-attention module, each modal feature in the modal triple is processed with queries, keys, and values, and the pairwise association weights within each modality and between modalities are calculated in parallel to capture complementary and conflicting information across modalities globally.

[0048] In one embodiment, the feature vectors of the three modalities at the same time (video modality sentiment features, audio modality sentiment features, and text modality sentiment features) are combined into a modality triple and fed into a multi-head cross-attention module. Through this module, each modality's feature simultaneously acts as a query, key, and value, and the pairwise association weights within and between modalities are calculated in parallel. In other words, the multi-head cross-attention mechanism can dynamically determine whether facial expressions or the words flooding the screen at the current moment are more noteworthy, thereby adaptively capturing complementary and conflicting information across modalities globally.

[0049] Furthermore, a pre-defined gated fusion network is used to normalize the three modal features in the modal triple after cross-modal interaction, obtaining normalized contribution coefficients. Then, based on the normalized contribution coefficients, the video modal sentiment features, audio modal sentiment features, and text modal sentiment features are weighted and synthesized to output a fused sentiment feature sequence that evolves over time.

[0050] In one embodiment, after the interaction is complete, a gated fusion network is introduced. This network dynamically generates normalized contribution coefficients for the three modalities based on the projected original features. These coefficients change in real time with the live broadcast content (e.g., audio has a higher weight when explaining a product, video has a higher weight when showcasing effects, and text has a higher weight when conducting interactive Q&A). Finally, the three modal features are weighted and synthesized according to these coefficients, outputting a fused sentiment feature sequence that evolves over time, achieving a qualitative leap from single-modal disparate statements to multi-modal mutual corroboration.

[0051] In one embodiment, in temporal cross-attention fusion, a shared attention mechanism can also be used. Instead of calculating all 6 pairs of cross-attentions, a unified fusion feature is calculated: Let the query Q = W_Q·[f_v(t), f_a(t), f_t(t)] (linear projection after concatenation), and the key values ​​K / V come from each modality. Then, cross-attention is calculated independently for each modality, and then summed. However, to maintain creativity, we use multi-head cross-attention, where the number of attention heads is 8. Each head calculates the pairwise relationships between modalities, and s is implemented through parallel group convolution. f_v, f_a, f_t are stacked into a matrix F ∈ R^{3×512}. Multi-head cross-attention is calculated: Attention(Q, K, V), where Q=F·W_Q, K=F·W_K, V=F·W_V. The output is the fused cross-feature f_cross ∈ R^{512}, which captures all pairwise interactions between modalities simultaneously.

[0052] S104. Perform trust-based global context representation mapping on the fused emotional feature sequence to obtain the real-time trust score of the live broadcast room in the current period.

[0053] It should be noted that although the fused features are rich in emotional information, there is a non-linear mapping relationship between emotional states (such as excitement and skepticism) and consumer trust (purchase intention), and trust is cumulative and time-varying. Therefore, a deep regression network with time-series awareness is needed to complete this transformation.

[0054] Specifically, the fused sentiment feature sequence is first input into a pre-defined deep mapping network. This deep mapping network possesses temporal awareness capabilities. Then, through multiple layers of nonlinear transformations within the deep mapping network, the fused sentiment features in the fused sentiment feature sequence are projected from the semantic space to the trust evaluation space.

[0055] Furthermore, through the temporal attention mechanism introduced by deep mapping networks, each time node in the fused emotional feature sequence within the trust evaluation space is subjected to quantitative evaluation processing based on expert rules, and key moments affecting trust are focused as global contextual representation vectors. These key moments include at least: moments when the broadcaster's emotions are intense and moments when questioning words appear in concentrated bullet comments.

[0056] In one embodiment, a deep mapping network with temporal awareness is first constructed. The input to this deep mapping network is a sequence of fused sentiment features, and the output is the trust score of the live stream within the current period. That is, the deep mapping network gradually projects sentiment features from the semantic space to the trust evaluation space through multiple layers of nonlinear transformations. However, in this process, not all historical sentiment information has equal value for the current trust judgment: certain key events (such as changes in the streamer's emotions or a concentration of questioning words in the chat) may play a decisive role in trust formation, while sentiment information from most stable periods contributes only to a limited extent. Therefore, the network introduces a temporal attention mechanism in its top-level design. This mechanism does not rely on fixed time windows or simple pooling operations, but instead uses learnable attention weights to automatically evaluate the importance of each time node in the fused feature sequence, focusing key moments that significantly influence the trust trend as global contextual representation vectors.

[0057] Furthermore, the global context representation vector needs to be mapped to a continuous scalar value through the regression layer of a deep mapping network, and then constrained to a preset interval by a nonlinear activation function to obtain the real-time trust score of the live stream in the current period. The real-time trust score reflects the consumer's overall trust level in the streamer and the live stream at the current moment.

[0058] In one embodiment, after obtaining the global context representation vector, the deep mapping network maps it to a continuous scalar value through a regression layer, and then constrains it to a preset interval through a nonlinear activation function to form the final trust score, which is the real-time trust score. The level of this score directly reflects the consumer's overall trust level towards the streamer and the live broadcast room at the current moment.

[0059] As a feasible implementation method, to further enhance the usability of the score evaluation results in actual deployment, it is also necessary to synchronously learn the variance estimate of the output value during the network training phase. This allows the deep mapping network to output the confidence score and its confidence interval simultaneously with only a single forward propagation during inference, without additional sampling or multiple inferences, thus ensuring both the reliability of the evaluation and maintaining real-time responsiveness.

[0060] S105. Conduct risk warnings and dynamic assessments based on the real-time trust score to obtain the final assessment result.

[0061] Specifically, the real-time trust score needs to be adaptively smoothed first. Then, based on the instantaneous fluctuation of the real-time trust score, the weight ratio between the accumulated historical score and the current observed score is dynamically adjusted to obtain the smoothed real-time trust score.

[0062] In one embodiment, after obtaining the real-time trust score, it needs to be transformed into an executable operational intervention instruction. This requires adaptive smoothing filtering of the original trust score sequence, and then dynamically adjusting the weight ratio of historical accumulation and current observations based on the instantaneous fluctuation of the real-time trust score. This approach allows for rapid response to sudden trust crises while avoiding false alarms triggered by extreme noise.

[0063] Furthermore, it is necessary to obtain several real-time trust scores within a fixed time window and determine them as a real-time trust score sequence. That is, within a fixed window period, obtain real-time trust scores at multiple different times and then arrange them in order into a real-time trust score sequence.

[0064] Furthermore, if the real-time trust score sequence shows a continuous upward trend, the final assessment result corresponding to the fixed time window period will be determined as a normal monitoring state. If the real-time trust score sequence shows a continuous downward trend but is within a controllable decline range, the final assessment result corresponding to the fixed time window period will be determined as a state requiring attention. If the real-time trust score sequence shows a continuous downward trend but is within a range of excessive short-term decline, the final assessment result corresponding to the fixed time window period will be determined as a state requiring mild intervention. If the real-time trust score sequence shows a continuous downward trend but is within a range of excessive long-term decline, the final assessment result corresponding to the fixed time window period will be determined as a state requiring mandatory intervention.

[0065] In one embodiment, a smoothed real-time trust score sequence is used for tiered early warning. Based on a preset trust threshold range, combined with the duration of the score and the single-cycle drop, the current live broadcast status can be divided into four levels: normal monitoring, attention alert, mild intervention, and mandatory intervention. Different suggested actions are then assigned to each level to obtain the final evaluation result corresponding to different real-time trust score sequences.

[0066] Furthermore, the final evaluation results are encapsulated and output as structured data, resulting in standardized evaluation structured data. This structured data includes: live stream status, timestamp, trust score, confidence interval, warning level, and contribution weight of each modality. In other words, the system encapsulates the calculated trust score, confidence interval, contribution weight of each modality, and warning level into a standardized structured data interface and outputs it externally. This data is then consumed in real-time by the e-commerce platform's credit rating system, intelligent recommendation ranking, or customer service dispatch center, thus completing a full technical loop from signal acquisition to value output.

[0067] In addition, embodiments of this application also provide a live streaming trust assessment system based on multimodal temporal emotion fusion, such as... Figure 2As shown, the live streaming trust assessment system 200 has instructions 202 that can be executed by the processor 201, enabling at least one processor 201 to execute: The video stream, audio stream, and bullet screen text stream collected in the live broadcast room are time-series aligned and preprocessed to obtain multimodal feature data. By using a pre-defined temporal encoder, temporal sentiment features are extracted from the multimodal feature data to obtain multimodal sentiment features. These multimodal sentiment features include video modal sentiment features, audio modal sentiment features, and text modal sentiment features. By sharing a semantic space, cross-modal temporal fusion processing based on multi-head cross-attention is performed between each modal sentiment feature in the multimodal sentiment features to obtain the fused sentiment feature sequence. The fused sentiment feature sequence is processed by global context representation mapping based on trust level to obtain the real-time trust score of the live broadcast room in the current period; The real-time trust score is used for risk warning and dynamic assessment to obtain the final assessment result.

[0068] This application extracts deep emotional features from the anchor's facial micro-expressions, vocal rhythm fluctuations, and real-time consumer feedback text. This allows trust assessment to be based on the mutual corroboration of multi-source information, avoiding the risk of misjudgment when single-modal signals are ambiguous in emotional expression or when data is sparse. Furthermore, the trust score is output in a fixed-period rolling fashion, ensuring that the assessment results are synchronized with the live broadcast process in real time. The dynamic tracking capability allows the platform to receive early warning signals when trust shows a downward trend. It can also directly map temporally fused emotional features into continuous trust scores, enabling quantitative assessment of emotional signals. Simultaneously, a multi-head cross-attention technique is introduced, allowing features from different modalities to interact in parallel pairs within a shared semantic space. This facilitates the dynamic capture of complementary cues between modalities and improves the trust prediction rate.

[0069] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0070] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0071] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0072] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0073] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0074] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0075] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0076] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0077] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0078] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0079] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of this specification.

Claims

1. A live streaming trust assessment method based on multimodal temporal sentiment fusion, characterized in that, The method includes: The video stream, audio stream, and bullet screen text stream collected in the live broadcast room are time-series aligned and preprocessed to obtain multimodal feature data. The multimodal feature data is processed by a preset temporal encoder to extract temporal sentiment features, thereby obtaining multimodal sentiment features; wherein, the multimodal sentiment features include: video modal sentiment features, audio modal sentiment features, and text modal sentiment features; By sharing a semantic space, cross-modal temporal fusion processing based on multi-head cross-attention is performed on each modal sentiment feature in the multimodal sentiment features to obtain a fused sentiment feature sequence. The fused emotional feature sequence is subjected to global context representation mapping based on trust level to obtain the real-time trust score of the live broadcast room in the current period; The real-time trust score is used for risk warning and dynamic assessment to obtain the final assessment result.

2. The live streaming trust assessment method based on multimodal temporal sentiment fusion according to claim 1, characterized in that, The video stream, audio stream, and bullet screen text stream collected in the live broadcast room are subjected to time-series alignment and preprocessing of multi-source live broadcast data to obtain multimodal feature data, specifically including: The multi-source live streaming data is obtained by pulling the video stream, audio stream, and bullet screen text stream in the current live streaming window in parallel through the live streaming platform interface. Based on the time anchor points of the video frames, the audio signals in the audio stream are divided into short segments corresponding to the duration of the video frames in the video stream according to the time axis, and the text in the bullet screen text stream is aggregated into a paragraph sequence according to a fixed time window to obtain the data time sequence alignment result of the multi-source live broadcast data; Based on the data timing alignment results, a timing-aligned multimodal data frame group is obtained; Data features of different modal data in the multimodal data frame group are extracted to obtain the multimodal feature data; wherein, the multimodal feature data includes the anchor's facial image features, speech acoustic signal features and consumer text feedback features at the same time.

3. The live streaming trust assessment method based on multimodal temporal sentiment fusion according to claim 1, characterized in that, The multimodal feature data is processed by a preset temporal encoder to extract temporal sentiment features, specifically including: By using a temporal encoder equipped with a three-dimensional spatiotemporal convolutional network, the video modal features in the multimodal feature data are subjected to joint convolution processing of facial change features and head pose shift features. Based on the spatial features of a single frame and the motion trend between adjacent frames, the video modal emotional features reflecting the anchor's real-time emotional state are output. By using a temporal encoder with a temporal recursive network, attention-focusing control is performed on the audio modal features in the multimodal feature data regarding the prosodic variation features. Based on the temporal sequence dependency of the temporal recursive network, the local audio segments with the most intense emotions in the audio modal features are automatically focused out, compressed, and the audio modal emotional features are output. The pre-trained language model is used to identify the contextual semantic representation of the text modal features in the multimodal feature data, and the identified contextual semantic representation is input into the temporal recursive network to capture the emotional evolution context features between multiple bullet screen texts. The emotional evolution context features are dynamically weighted with the preset emotional word intensity to output the text modality emotional features with high emotional sensitivity in consumer feedback.

4. The live streaming trust assessment method based on multimodal temporal sentiment fusion according to claim 1, characterized in that, By sharing a semantic space, cross-modal temporal fusion processing based on multi-head cross-attention is performed on the sentiment features of each modality in the multimodal sentiment features to obtain a fused sentiment feature sequence, specifically including: By using a feedforward projection network, each modal sentiment feature in the multimodal sentiment features is mapped to the shared semantic space of the same dimension; The video modal sentiment features, audio modal sentiment features, and text modal sentiment features at the same time are combined into modal triples and input into the multi-head cross-attention module; The multi-head cross-attention module performs queries, keys, and values ​​on each modal feature in the modal triplet, and calculates the pairwise association weights within and between each modality in parallel, so as to capture complementary and conflicting information across modalities globally. By using a pre-defined gated fusion network, the three modal features in the modal triple after cross-modal interaction are normalized to obtain the normalized contribution coefficient. Based on the normalized contribution coefficient, the video modal sentiment features, the audio modal sentiment features, and the text modal sentiment features are weighted and synthesized to output a fused sentiment feature sequence that evolves over time.

5. The live streaming trust assessment method based on multimodal temporal sentiment fusion according to claim 1, characterized in that, The fused sentiment feature sequence is subjected to trust-based global context representation mapping to obtain the real-time trust score of the live broadcast room in the current period, specifically including: The fused emotional feature sequence is input into a preset deep mapping network; wherein the deep mapping network has time-series awareness capability; Through multi-layer nonlinear transformations in the deep mapping network, the fused sentiment features in the fused sentiment feature sequence are projected from the semantic space to the trust evaluation space; Through the temporal attention mechanism introduced by the deep mapping network, each time node in the fused emotional feature sequence in the trust evaluation space is subjected to quantitative evaluation processing based on expert rules, and the key moments affecting trust are focused as global context representation vectors; wherein, the key moments include at least: the moment when the anchor's emotions are intense and the moment when questioning words appear in the barrage. The global context representation vector is mapped to a continuous scalar value through the regression layer of the deep mapping network, and constrained to a preset interval by a nonlinear activation function to obtain the real-time trust score of the live broadcast room in the current period; wherein, the real-time trust score reflects the consumer's overall trust level in the anchor and the live broadcast room at the current moment.

6. The live streaming trust assessment method based on multimodal temporal sentiment fusion according to claim 1, characterized in that, Before applying the real-time trust score for risk warning and dynamic assessment to obtain the final assessment result, the method further includes: The real-time trust score is subjected to adaptive smoothing filtering. Based on the instantaneous fluctuation range of the real-time trust score, the weight ratio between the historical accumulated score and the current observed score is dynamically adjusted to obtain the smoothed real-time trust score.

7. The live streaming trust assessment method based on multimodal temporal sentiment fusion according to claim 6, characterized in that, The real-time trust score is used for risk warning and dynamic assessment to obtain the final assessment result, which specifically includes: Obtain several real-time trust scores within a fixed time window and determine them as a real-time trust score sequence; If the real-time trust score sequence is continuously increasing, then the final evaluation result corresponding to the fixed time window period is determined as the normal monitoring state; If the real-time trust score sequence is continuously decreasing and within a controllable decreasing range, then the final evaluation result corresponding to the fixed time window period will be determined as a state requiring attention. If the real-time trust score sequence is continuously declining and is in a short-term excessive decline range, then the final evaluation result corresponding to the fixed time window period will be determined as a state requiring mild intervention. If the real-time trust score sequence is continuously declining and is in a long-term excessive decline range, then the final evaluation result corresponding to the fixed time window period will be determined as a state requiring mandatory intervention.

8. The live streaming trust assessment method based on multimodal temporal sentiment fusion according to claim 7, characterized in that, After obtaining the final evaluation result, the method further includes: The final evaluation results are encapsulated and output as structured data to obtain standardized evaluation structure data; wherein, the evaluation structure data includes: live room identification status, timestamp, trust score, confidence interval, warning level, and contribution weight of each modality.

9. The live streaming trust assessment method based on multimodal temporal sentiment fusion according to claim 1, characterized in that, The shared semantic space includes a video projection layer, an audio projection layer, and a text projection layer. Each projection layer consists of two fully connected layers, a residual connected layer, and a layer normalization layer.

10. A live streaming trust assessment system based on multimodal temporal emotion fusion, characterized in that, The live streaming trust assessment system has instructions that can be executed by a processor to enable the processor to perform a live streaming trust assessment method based on multimodal temporal emotion fusion according to any one of claims 1-9.

Images