An e-commerce live broadcast real-time traffic prediction method and system based on multi-modal learning

Abstract

The application discloses an e-commerce live broadcast real-time traffic prediction method and system based on multi-modal learning, comprising the following steps: constructing a host multi-modal complementary attention network, combining LSTM to learn the aligned host space-time multi-modal representation and the audience multi-variable time sequence representation; constructing a causal mask cross-attention network to learn the audience representation fused with the host information and the host representation fused with the audience information, fusing the two through an adaptive attention network to obtain a live broadcast joint interaction state sequence; decoupling the live room real-time traffic into entry traffic and exit traffic, respectively performing time sequence modeling, taking the live room joint interaction state sequence as a conditional signal, respectively enhancing the entry traffic and exit traffic representation through causal cross-attention, and fusing the two to predict the traffic peak value in a future period of time. The application can realize high-precision early prediction of the live broadcast traffic peak value, and effectively guide the dynamic adjustment of the host live broadcast strategy.

Description

Technical Field

[0001] This invention relates to the field of e-commerce technology, and in particular to a method and system for real-time traffic prediction in e-commerce live streaming based on multimodal learning. Background Technology

[0002] With the rapid development of live-streaming e-commerce, it has become a key channel for online retail. However, the highly real-time and interactive nature of live-streaming e-commerce leads to dramatic and unpredictable fluctuations in traffic metrics such as the number of online users. This high degree of uncertainty in user traffic poses a significant challenge to the refined operation of both streamers and platforms. Therefore, accurately predicting short-term traffic changes based on the real-time status of the live-streaming room is crucial for guiding streamers to dynamically adjust their presentation pace, interaction strategies, and product promotion timing to maximize commercial value.

[0003] Currently, existing live streaming traffic prediction methods have significant technical limitations. For example, some methods rely primarily on traditional statistical or multivariate models. These methods either extrapolate from single time-series data of historical online viewership or, while integrating multi-dimensional behavioral data such as user clicks and purchases, still fundamentally depend on structured numerical features. They struggle to capture sudden traffic spikes caused by unstructured content such as the streamer's presentation, emotional state, and body language, resulting in bottlenecks in prediction accuracy and responsiveness to unexpected situations. To address these issues, other existing technologies attempt to introduce multimodal analysis methods, incorporating visual and audio information from the streamer and text comments from the audience into the model. However, when performing modal fusion, these methods typically treat information from different modalities as independent feature sources, simply splicing or processing them in parallel. This fails to effectively model and quantify the real-time, dynamic causal interaction between streamer behavior (such as scripted guidance and product display) and audience feedback (such as comments / questions and likes), leading to insufficient depiction of the real interactive atmosphere in the live stream and thus limiting further improvements in prediction performance.

[0004] In fact, e-commerce live streaming is a typical multimodal information scenario. Fluctuations in its traffic are the result of multiple factors, including the host's audio and video information (such as language, emotions, and actions), product explanations, and user interactions. Existing technologies generally fail to effectively model the real-time interactions of this heterogeneous multimodal data for comprehensive analysis. This results in an inability to fully and accurately depict the real-time dynamics of the live stream, and the accuracy and reliability of predictions are insufficient to meet the needs of real-time decision-making. Therefore, a new technical solution is urgently needed in this field to address these problems. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a method and system for real-time traffic prediction in e-commerce live streaming based on multimodal learning. This solution aims to address one or more of the following problems in existing technologies: 1) Existing traffic prediction models do not fully utilize unstructured content information such as video and audio within the live streaming room, resulting in insensitivity to sudden changes in traffic caused by the host's real-time behavior and explanations, leading to low prediction accuracy. 2) Existing multimodal analysis methods are relatively simple at the information fusion level, failing to deeply explore and characterize the intrinsic relationships between different modal information, and have limited feature representation capabilities. 3) Existing technologies generally lack effective modeling of the core element of e-commerce live streaming scenarios—namely, the real-time dynamic interaction between the host and the audience—and cannot accurately capture the collaborative traffic changes brought about by the interaction.

[0006] To achieve the above objectives, this invention provides a method for real-time traffic prediction in e-commerce live streaming based on multimodal learning, aiming to solve the problems in existing technologies, such as the simplistic strategy for fusion of multimodal information in live streaming prediction models and the lack of modeling for the dynamic interaction relationship between the streamer and the audience.

[0007] In a first aspect, the present invention provides a method for real-time traffic prediction in e-commerce live streaming based on multimodal learning, comprising the following steps: S1, collecting live video streams and user behavior data, extracting multimodal representations of the streamer and temporal representations of the audience, and then performing interactive... The supplementary attention mechanism dynamically fuses complementary information from various modalities to obtain a spatiotemporal multimodal representation of the anchor. S2. Based on the spatiotemporal multimodal representation of the anchor and the temporal representation of the audience, the bidirectional dynamic interaction between the anchor and the audience is modeled by causal cross-attention, and the joint interaction state sequence of the live room is obtained by adaptive gating fusion. S3. Decouple the incoming traffic and outgoing traffic into two streams, perform time-series modeling on each, and enhance the dual-stream representation through causal cross-attention using the joint interaction state sequence of the live room as a conditional signal. After fusion, predict the future traffic peak.

[0008] Further, step S1 specifically includes: S11. Collect raw live data in units of a preset sliding time window. The raw live data includes live video stream and user behavior data. The user behavior data includes dynamic traffic characteristics and real-time interaction characteristics. S12. Perform frame extraction and audio separation on the live video stream, and extract feature representations of four modalities: text, visual, audio, and action; perform standardization processing on the user behavior data to generate a user behavior feature sequence; S13. Temporally align the modal representations and user behavior features using a Long Short-Term Memory (LSTM) network. For any target modality, treat the remaining modalities as a set of complementary modalities. Calculate the correlation between the target modality and each complementary modality using the Hadamard product. After softmax normalization, subtract all-one vectors to obtain complementary weights. Weight the complementary modalities to obtain non-redundant compensation features. Summate these features to obtain the compensation representation of the target modality. Iterate through all time steps to obtain the compensation representation sequence for each modality. S14. For each target modality, its alignment representation is used as the query and the compensation representation as the key and value. After causal cross-attention interaction, it is added to the original alignment representation and then causal self-attention aggregation to obtain the final fusion representation of the modality. The final fusion representations of all modalities are spliced ​​together to obtain the broadcaster's spatiotemporal multimodal representation. S15. The user behavior feature sequence is passed through a long short-term memory network to obtain a multivariate temporal representation of the audience.

[0009] Further, step S2 specifically includes: S21. For the anchor representation sequence, obtain the anchor-side historical representation through causal self-attention, and obtain the anchor-side interactive representation through causal cross-attention with the anchor representation as query and the audience representation as key and value; the causal cross-attention refers to using a causal mask matrix to shield the information of future time steps in the attention calculation, so that the attention calculation of the current time step only depends on the information of the current and historical time steps. S22. For the audience representation sequence, obtain the audience-side historical representation through causal self-attention, and obtain the audience-side interactive representation through causal cross-attention with the audience representation as the query and the anchor representation as the key and value. S23. Adaptively weightedly fuse the broadcaster's historical representation and interaction representation at each time step using sigmoid gating to obtain the updated broadcaster's state representation; perform the same operation on the viewer's side to obtain the updated viewer's state representation. S24. The updated state representations of the broadcaster and the audience at each time step are spliced ​​together to obtain the joint interactive state sequence of the live broadcast room.

[0010] Further, step S3 specifically includes: S31. Decouple the inbound flow characteristics and outbound flow characteristics, and input them into two independent long short-term memory networks for time series modeling to obtain inbound representation sequences and outbound representation sequences. S32. Using the joint interaction state sequence of the live room as the key and value, and using the entry representation sequence and the exit representation sequence as the query respectively, the two representation sequences are enhanced by causal cross-attention. S33. The enhanced entry and exit characterization sequences are spliced ​​together, fused through a feedforward layer, and then output through a regression layer as the predicted peak traffic value within a preset time period in the future. The peak traffic value is the highest instantaneous number of online users within the preset prediction window.

[0011] Specifically, the live video stream is frame-sampling using multi-rate sampling. A first sampling rate is used to extract global visual frames to capture the overall image information of the live room, and a second sampling rate higher than the first sampling rate is used to extract action frames to capture changes in the anchor's body movements. The first sampling rate is 1-2 frames / second, and the second sampling rate is 8-15 frames / second. The action frames are identified and extracted based on the pixel differences between adjacent frames exceeding a preset threshold, where the preset threshold is the root mean square of the difference in grayscale values ​​of corresponding pixels in adjacent frames.

[0012] Specifically, the original live data is sampled in units of a preset sliding time window, and the preset time window, time step and prediction window are preset values; the time window is 2.5 to 3.5 minutes, each time step is 3 to 6 seconds, and the prediction window is 0.5 to 1.5 minutes.

[0013] Specifically, the pre-trained networks corresponding to the four modalities include: VisionTransformer for the visual modality, S3D for the action modality, Librosa for extracting MFCCs features for the audio modality, and BERT for the language modality.

[0014] A second aspect of the present invention is a real-time traffic prediction system for e-commerce live streaming based on multimodal learning, the system comprising: The representation learning module is used to collect raw live data, extract multimodal representations of the anchor and temporal representations of the audience, and generate spatiotemporal multimodal representations of the anchor and multivariate temporal representations of the audience through complementary attention fusion. The main interaction module is used to obtain the joint interaction state sequence of the live broadcast room based on the host's spatiotemporal multimodal representation and the audience's multivariate temporal representation, through causal masking across attention and adaptive gating; The flow decoupling module is used to decouple the inbound and outbound flows into two streams, perform time-series modeling on them separately, and generate inbound and outbound characterization sequences. The traffic prediction module is used to predict future traffic peaks by using the joint interaction state sequence of the live room as a conditional signal, enhancing the entry and exit representation sequences through causal cross-attention respectively, and then fusing them.

[0015] A third aspect of the present invention provides an electronic device comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the aforementioned method for real-time traffic prediction in e-commerce live streaming based on multimodal learning.

[0016] In a fourth aspect, the present invention provides a computer-readable storage medium having stored thereon computer program instructions, which, when executed by a processor, implement the aforementioned method for real-time traffic prediction of e-commerce live streaming based on multimodal learning.

[0017] In this embodiment of the invention, the broadcaster's representation is learned through a multimodal fusion method, effectively bridging the gap between various modalities; cross-attention networks and adaptive attention networks are used to effectively simulate the interaction state between the broadcaster and the audience in the actual live broadcast room, and the state of the live broadcast room is modeled; the dynamic traffic features are decoupled, and the joint interaction state of the live broadcast room is added to the traffic prediction through cross-attention networks, so as to realize the prediction of traffic peaks in the future period of time and guide the dynamic adjustment of the broadcaster's live broadcast strategy. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the real-time traffic prediction method for e-commerce live streaming based on multimodal learning, as described in an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of the real-time traffic prediction system for e-commerce live streaming based on multimodal learning, according to an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0020] (I) Definition of Key Terms In this invention, "causal cross-attention" refers to using a causal mask matrix in attention calculation to shield information from future time steps, ensuring that attention calculation at the current time step relies only on information from the current and historical time steps. "Causal self-attention" refers to using a causal mask matrix in self-attention calculation to ensure that temporal modeling utilizes only current and historical information. This causal masking mechanism ensures that interaction modeling and temporal modeling at each time step do not depend on future information, avoiding information leakage during prediction.

[0021] like Figure 1 As shown in the figure, an embodiment of the present invention provides a method for real-time traffic prediction in e-commerce live streaming based on multimodal learning, which includes the following steps: S1: Collect raw live data and extract multimodal representations of the anchor and temporal representations of the audience.

[0022] This step extracts the broadcaster's multimodal features from the video using a pre-trained network; it then constructs a multimodal complementary attention network mechanism for the broadcaster, and learns aligned spatiotemporal multimodal representations of the broadcaster and multivariate temporal representations of the audience based on the constructed multimodal network and long short-term memory network, respectively. These spatiotemporal multimodal representations of the broadcaster and multivariate temporal representations of the audience serve as the fundamental inputs for broadcaster-audience interaction modeling in subsequent steps.

[0023] (1.1) Collect raw live broadcast data: Raw live streaming data is collected in units of a preset sliding time window. The raw live streaming data includes the live video stream and user behavior data. The user behavior data includes dynamic traffic features and real-time interaction features. The dynamic traffic features include at least the number of users online in real-time, the number of users entering within the preset time window, and the number of users leaving. The real-time interaction features include at least the number of likes, comments, shares, and favorites. The raw live streaming data is sampled in units of a preset sliding time window. Preferably, the time window is 2.5–3.5 minutes, each time step is 3–6 seconds, and the prediction window is 0.5–1.5 minutes. The above parameter ranges were determined through experimental optimization: when the time window is shorter than 2.5 minutes, the model struggles to capture sufficient temporal dependency information; when the time window is longer than 3.5 minutes, the model's computational overhead increases significantly and redundant information is introduced. It should be noted that the above endpoint values ​​are all included within the scope of protection claimed by this invention.

[0024] In a specific sampling example, the raw data is first sampled in a sliding window at 3-minute intervals to form a sample sequence. Each time step is a specific sampling time window, which is 5 seconds in this case. The observation window is 2.5 minutes, and the prediction window is 0.5 minutes. That is, each time the latest 2.5 minutes of live stream content is used to predict the traffic peak in the live stream in the next 0.5 minutes.

[0025] (1.2) Extracting multimodal feature representations of broadcasters and user behavior feature sequences: The live video stream is subjected to frame extraction and audio separation. Specifically, multi-rate sampling is used: global visual frames are extracted at a first sampling rate (1-2 frames / second) to capture the overall picture information of the live room; motion frames are extracted at a second sampling rate (8-15 frames / second) higher than the first sampling rate to capture the changes in the anchor's body movements; the motion frames are identified and extracted based on the pixel difference between adjacent frames exceeding a preset threshold, the preset threshold being the root mean square of the difference in grayscale values ​​of corresponding pixels in adjacent frames, with a threshold range of 25-35.

[0026] For the image frames obtained by frame extraction and the separated audio streams, feature representations for four modalities—text, vision, audio, and action—are extracted using pre-trained networks. The pre-trained networks corresponding to these four modalities include: VisionTransformer (ViT-B / 16) for the vision modality, S3D (Kinetics pre-trained weights) for the action modality, MFCCs features extracted using the Librosa library for the audio modality (40-dimensional, 25ms window, 10ms stride), and BERT-base (12 layers, 768 dimensions) for the language modality. It should be noted that the above-mentioned pre-trained networks are only preferred implementations of this invention. In practical applications, they can be replaced with other equivalent pre-trained models depending on computing power and accuracy requirements. Such replacements do not constitute a substantial change to the technical solution of this invention.

[0027] Simultaneously, the user behavior data is standardized to generate a user behavior feature sequence. Specifically, the Z-score standardization method is used to calculate the mean and standard deviation of each feature dimension (including real-time online users, number of users entering / leaving, number of likes, number of comments, etc.) within the observation window, converting the original values ​​into standard scores with a mean of 0 and a standard deviation of 1, in order to eliminate the differences in units and orders of magnitude between different features, making model training more stable and efficient.

[0028] (1.3) Temporal alignment of modal representations and user behavior features is performed using a long short-term memory network, and the compensated representation sequences for each modality are obtained through a complementary attention mechanism: Based on the sequence representations of the above modalities, the representation dimensions of each time step are first time-aligned using a Long Short-Term Memory (LSTM) network; the representation dimensions of each modality at each time step are then unified to facilitate subsequent cross-modal fusion.

[0029] Based on the above-mentioned aligned four modal sequence representations (text modality, visual modality, audio modality, and action modality), for any target modality, the other three modalities constitute a complementary modality set for that target modality; specifically, for each time step... The representation vector of the target mode is calculated using the Hadamard product. The representation vector of each complementary mode in the complementary mode set The Hadamard product between them, and the results are analyzed. Normalization yields the feature-level correlation vector of the complementary mode relative to the target mode, calculated as follows: in, Indicates the target mode in the th... The representation vector at each time step, Indicates the first The complementary mode in the th... The representation vector at each time step, This indicates that the Hadamard product operation is an element-wise multiplication. The normalization function is used to make the sum of the elements of the resulting correlation vector equal to 1. The obtained feature-level correlation vector; each element in the correlation vector reflects the degree of information redundancy between two modes on the corresponding feature dimension, and the larger the correlation value, the higher the cross-modal redundancy on that dimension.

[0030] Subtract the correlation vector from the all-1 vector to obtain the complementary weight vector; use the complementary weight vector to perform element-wise weighting on the representation vector of the corresponding complementary mode to obtain the non-redundant compensation features contributed by each complementary mode; sum the non-redundant compensation features contributed by all complementary modes to obtain the compensation vector of the target mode at this time step; traverse all time steps to obtain the compensation representation sequence of the target mode; perform the above operations on the four modes respectively to obtain the compensation representation sequence of each mode.

[0031] (1.4) Obtain the final fusion representation of each modality through causal cross-attention and causal self-attention, and splice them together to obtain the spatiotemporal multimodal representation of the anchor: For each target modality, the aligned representation sequence of the target modality is used as the query, and its compensated representation sequence is used as the key and value. Cross-modal interaction is performed through a causal cross-attention mechanism to obtain the cross-modal interaction representation. The original aligned representation of the target modality is added to the cross-modal interaction representation, and the temporal dependencies are further aggregated through a causal self-attention layer to obtain the final fusion representation of the target modality. The above operations are performed on the four modalities respectively, and the final fusion representations of each modality are concatenated to obtain the broadcaster's spatiotemporal multimodal representation.

[0032] The causal cross-attention refers to using a causal mask matrix in attention calculation to mask information from future time steps, so that attention calculation at the current time step only depends on information from the current and historical time steps; the causal self-attention refers to using a causal mask matrix in self-attention calculation to ensure that time series modeling only utilizes current and historical information.

[0033] (1.5) The user behavior feature sequence is passed through a long short-term memory network to obtain a multivariate temporal representation of the audience: The aligned user behavior feature sequence from step (1.3) is input into a long short-term memory network (LSTM) to further capture the temporal dependencies between various behavioral features on the audience side and obtain a multivariate temporal representation sequence of the audience. This representation sequence depicts the behavioral evolution of the audience group within the observation window and is used for subsequent anchor-audience interaction modeling.

[0034] In one specific embodiment, assuming a five-second time step, user behavior features include multiple variables such as real-time online users, number of users entering, number of users leaving, number of likes, number of comments, number of reposts, and number of favorites. After LSTM modeling, a fixed-dimensional multivariate temporal representation sequence of the audience is output. At each time step, the visual representation is processed by the pre-trained model VisionTransformer with dimensions of (5, 1000), the action representation is processed by the pre-trained model S3D with dimensions of (1024), the video representation is processed by the Librosa model to extract MFCCs features with dimensions of (216, 10), and the language representation is processed by the pre-trained model BERT with dimensions of (768). For the two-dimensional representations of vision and language, they are converted into one dimension by flattening, and then processed by a long short-term memory network, for example, converting all modalities into 256-dimensional representations at each time step, which facilitates the fusion between modalities after alignment. It should be noted that the above-mentioned pre-trained network is only a preferred implementation of the present invention. In practical applications, it can be replaced with other equivalent pre-trained models according to the computing power conditions and accuracy requirements. Such replacement does not constitute a substantial change to the technical solution of the present invention.

[0035] In one embodiment, in a practical application on an e-commerce platform, when the method of the present invention performs interpretability analysis through a visualized attention weight matrix, it was found that the weights of each modality show a relatively consistent trend of change. Among them, the visual modality is more responsive, while language text shows a certain delay compared to the visual modality, indicating that users are more sensitive to visual features.

[0036] S2: Based on the spatiotemporal multimodal representation of the anchor and the multivariate temporal representation of the audience, construct the anchor-audience dynamic interaction network to obtain the joint interaction state sequence of the live broadcast room.

[0037] Compared to existing technologies that treat different modal information as independent feature sources and simply splice or process them in parallel, this invention constructs a two-way interactive network between the broadcaster and the audience to effectively model and quantify the real-time dynamic causal interaction between the broadcaster's behavior and the audience's feedback, thereby obtaining a joint interactive state sequence that depicts the real interactive atmosphere of the live broadcast room.

[0038] The aforementioned anchor-viewer dynamic interaction network employs a causal masking mechanism. During attention calculation, a masking matrix is ​​used to shield information from future time steps, ensuring that the interaction modeling at each time step relies solely on current and historical information, thus preventing the leakage of future information.

[0039] (2.1) Obtaining the broadcaster's historical representation and the broadcaster's interactive representation: The interactive network comprises two parallel interactive paths: First, the host's spatiotemporal multimodal representation sequence obtained in step (1.4) is modeled through a causal self-attention network to model the temporal continuity of the host's livestream rhythm and explanation strategies, resulting in a host self-attention representation sequence (i.e., the host's historical representation). Simultaneously, using the audience's multivariate temporal representation sequence as a query and the host's spatiotemporal multimodal representation sequence as a key and value, a causal cross-attention network is used to calculate the dynamic impact of the host's behavior on the audience's state, obtaining the host's interaction representation sequence with the audience. This path simulates the guiding effect of the host's explanations, actions, emotions, and other behaviors on the audience's viewing decisions.

[0040] (2.2) Obtaining audience-side historical representations and audience-side interaction representations: The audience multivariate temporal representation sequence obtained in step (1.5) is modeled through a causal self-attention network to model the temporal evolution of the audience's own behavior (such as the herd effect, interest decay, etc.) and obtain the audience self-attention representation sequence (i.e. audience-side historical representation).

[0041] Simultaneously, using the audience's multivariate temporal representation sequence as the query and the anchor's spatiotemporal multimodal representation sequence as the key and value, a causal cross-attention network is used to calculate the dynamic impact of the anchor's behavior on the audience's state, thereby obtaining the anchor's interaction representation sequence with the audience (i.e., audience-side interaction representation). This path simulates the guiding role of the anchor's explanations, actions, emotions, and other behaviors on the audience's viewing decisions.

[0042] (2.3) Adaptive weighted fusion is performed using sigmoid gating to obtain updated broadcaster-side and viewer-side state representations: For each time step, the broadcaster has two information sources: the broadcaster's self-attention representation, reflecting the continuity of their own live-streaming strategy, and the audience's interaction representation, reflecting the broadcaster's adjustments based on real-time audience feedback. The importance of these two sources changes dynamically across different live-streaming stages. For example, in the initial stages of product listing, the broadcaster relies more on their own strategy, while during peak interaction periods, they rely more on audience feedback. Therefore, an adaptive gating mechanism is introduced: For the broadcaster side, the broadcaster's self-attention representation and the audience's interaction representation with the broadcaster are weighted and summed using learnable gating weights to obtain the updated broadcaster-side state representation. For the audience side, the same weighted fusion operation is performed on the audience self-attention representation and the anchor's interaction representation with the audience to obtain the updated audience-side state representation. The gating weights are dynamically generated by the features of the current time step through the sigmoid activation function, enabling the model to adaptively adjust the relative proportion of its historical inertia and cross-entity interaction feedback.

[0043] (2.4) Concatenate to obtain the joint interactive state sequence of the live broadcast room: By concatenating the updated anchor-side state representation and the audience-side state representation along the feature dimension at each time step, the joint interaction representation of the live room at that time step is obtained; by traversing all time steps, the joint interaction state sequence of the live room is obtained.

[0044] S3: Decouple the real-time traffic characteristics of the live room into inbound traffic and outbound traffic, perform time-series modeling on each, and use the joint interaction state sequence of the live room as the interaction condition signal for feature enhancement, fuse the two streams (inbound traffic and outbound traffic) and predict future traffic peaks.

[0045] Because inbound and outbound traffic respond to the live stream status at fundamentally different speeds—outbound traffic reflects the real-time decisions of the current audience, is more sensitive to changes in the streamer's behavior and the interactive atmosphere, and has a shorter response lag; inbound traffic, on the other hand, is more influenced by the platform's recommendation system and exposure outside the live stream, with a certain delay in dissemination and decision-making from exposure to actual entry, resulting in a longer response lag—this step decouples the two types of traffic and models them separately.

[0046] In this step, the joint interaction state sequence of the live room, i.e. the output result of step (2.4), serves as the input source of the interaction condition signal in this step.

[0047] (3.1) Decouple the inflow and outflow characteristics and input them into two independent LSTMs for time series modeling to obtain the inflow and outflow characterization sequences: The inbound and outbound traffic features obtained in step (1.1) are decoupled into two independent traffic flows at the feature layer; specifically, the number of inbound users and the number of outbound users at each time step are separated from the dynamic traffic features to form an inbound traffic feature sequence and an outbound traffic feature sequence, respectively.

[0048] The inbound and outbound flow feature sequences are input into two independent Long Short-Term Memory (LSTM) networks for temporal dependency modeling, yielding inbound and outbound representation sequences respectively. The two LSTM networks share the same network structure but not the same parameters, allowing them to capture the distinct temporal evolution patterns of the two flow types.

[0049] (3.2) Using the joint interaction state sequence of the live broadcast room as the key and value, and using the entry representation sequence and exit representation sequence as queries respectively, the two representation sequences are enhanced by causal cross-attention: The joint interaction state sequence of the live room obtained in step (2.4) is used as the interaction condition signal, which carries the dynamic interaction evolution information between the anchor and the audience.

[0050] Using the entry representation sequence as a query and the interaction condition signal as a key and value, a causal cross-attention mechanism is used to fuse the dynamic interaction context information of the live stream into the entry representation, resulting in an enhanced entry representation sequence. This operation enables entry traffic prediction to perceive the real-time interaction atmosphere of the current live stream, thereby capturing potential entry demands triggered by the live stream content. The same operation is performed on the departure representation sequence: using the departure representation sequence as the query and the interaction condition signal as the key and value, an interaction-enhanced departure representation sequence is obtained through a causal cross-attention mechanism. The interaction condition signal carries the dynamic interaction evolution information between the broadcaster and the audience modeled in step S2. By injecting it as a key and value into the traffic representation, each traffic stream can perceive the real-time interaction state of the live stream, thus providing an interaction perception context for traffic prediction. This operation enables departure traffic prediction to perceive the real-time interaction atmosphere of the current live stream, thereby capturing the departure risk caused by a decline in interactive experience or changes in content attractiveness.

[0051] (3.3) Dual-stream fusion and peak traffic prediction: The enhanced entry and exit characterization sequences are spliced ​​along the feature dimension, and the enhanced interaction features of the two traffic streams are further fused through a feedforward neural network layer. The peak traffic prediction value of the live room within a preset time period (preferably 0.5 to 1.5 minutes) is output through a regression prediction layer.

[0052] (II) Model Performance Validation To verify the technical effectiveness of the present invention, comparative experiments were conducted on live streaming datasets from multiple e-commerce platforms. The following specific embodiments illustrate these experiments.

[0053] Example 1: Prediction of Peak Traffic for All Product Categories This embodiment uses comprehensive livestream traffic data from an e-commerce platform, dividing the dataset into popular categories and other categories. The performance of the method described in this invention on normal traffic (samples where the peak traffic in the prediction window does not deviate from the mean traffic in the observation window by more than one standard deviation) is then tested. The evaluation metrics are root mean square error (RMSE) and mean absolute error (MAE), with lower values ​​indicating higher prediction accuracy.

[0054] The test results are as follows: (a) In popular categories: RMSE reached 28.143 and MAE reached 6.999. Compared with advanced multivariate time series models, RMSE improved by 17.1% and MAE improved by 15.4%; compared with advanced multimodal video prediction models, RMSE improved by 6.9% and MAE improved by 8.3%. (b) For the remaining categories, RMSE reached 4.902 and MAE reached 1.983. Compared with the advanced multivariate time series model, RMSE improved by 53.6% and MAE improved by 39.9%; compared with the advanced multimodal video prediction model, RMSE improved by 56.9% and MAE improved by 38.1%.

[0055] The above results demonstrate that the method of the present invention significantly improves the prediction accuracy for live streaming rooms of different categories under normal traffic prediction scenarios, with a more significant improvement for non-popular categories. Example 2: Abnormal Traffic Peak Prediction In this embodiment, abnormal flow is defined as a sample whose peak flow in the prediction window deviates from the mean flow in the observation window by at least one standard deviation. Sudden increases and decreases in flow are defined as deviations upward and downward by one standard deviation, respectively. The test conditions are the same as in Embodiment 1.

[0056] The test results are as follows: (a) Under popular categories: For sudden traffic surges, the RMSE reached 34.101 and the MAE reached 8.222. Compared with advanced multivariate time series models, the RMSE improved by 14.2% and the MAE improved by 11.9%; compared with advanced multimodal video prediction models, the RMSE improved by 2.9% and the MAE improved by 2.5%. For sudden traffic drops, the RMSE reached 24.566 and the MAE reached 7.990. Compared with advanced multivariate time series models, the RMSE improved by 35.3% and the MAE improved by 32.8%; compared with advanced multimodal video prediction models, the RMSE improved by 32.9% and the MAE improved by 33.2%.

[0057] (b) For other categories: For sudden traffic surges, the RMSE reached 5.985 and the MAE reached 2.302. Compared with advanced multivariate time series models, the RMSE improved by 51.6% and the MAE improved by 37.5%; compared with advanced multimodal video prediction models, the RMSE improved by 55.6% and the MAE improved by 36.1%. For sudden traffic drops, the RMSE reached 5.433 and the MAE reached 2.253. Compared with advanced multivariate time series models, the RMSE improved by 56.7% and the MAE improved by 45.2%; compared with advanced multimodal video prediction models, the RMSE improved by 63.9% and the MAE improved by 50.7%. The above results demonstrate that the method of this invention maintains excellent predictive performance in predicting abnormal scenarios such as sudden increases and decreases in traffic, effectively addressing abnormal traffic changes within live streaming rooms. Compared to normal traffic, abnormal traffic changes are more likely to attract the attention of broadcasters and require further strategy adjustments. The technical effects of this invention have significant guiding significance for actual live streaming operations.

[0058] like Figure 2 As shown, the present invention also provides a real-time traffic prediction system for e-commerce live streaming based on multimodal learning, the system comprising: The representation learning module is used to collect raw live data, extract multimodal representations of the anchor and temporal representations of the audience, and generate spatiotemporal multimodal representations of the anchor and multivariate temporal representations of the audience through complementary attention fusion. The main interaction module is used to obtain the joint interaction state sequence of the live broadcast room based on the host's spatiotemporal multimodal representation and the audience's multivariate temporal representation, through causal masking across attention and adaptive gating; The flow decoupling module is used to decouple the inbound and outbound flows into two streams, perform time-series modeling on them separately, and generate inbound and outbound characterization sequences. The traffic prediction module is used to predict future traffic peaks by using the joint interaction state sequence of the live room as a conditional signal, enhancing the entry and exit representation sequences through causal cross-attention respectively, and then fusing them.

[0059] Accordingly, this application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-described method for real-time traffic prediction of e-commerce live streaming based on multimodal learning. The hardware structure diagram of an e-commerce live streaming real-time traffic prediction system based on multimodal learning provided in this embodiment of the invention is shown in any data processing device. Besides the processor, memory, and network interface, the data processing device in this embodiment may also include other hardware depending on the actual function of the data processing device, which will not be elaborated further.

[0060] Accordingly, this application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the aforementioned multimodal learning-based real-time traffic prediction method for e-commerce live streaming. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0061] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for real-time traffic prediction in e-commerce live streaming based on multimodal learning, characterized in that, Including the following step: S1. Collect live video streams and user behavior data, extract multimodal representations of the anchor and temporal representations of the audience, and dynamically fuse complementary information from each modality through a complementary attention mechanism to obtain the spatiotemporal multimodal representation of the anchor. S2. Based on the spatiotemporal multimodal representation of the anchor and the temporal representation of the audience, the bidirectional dynamic interaction between the anchor and the audience is modeled by causal cross-attention, and the joint interaction state sequence of the live room is obtained by adaptive gating fusion. S3. Decouple the incoming traffic and outgoing traffic into two streams, perform time-series modeling on each, and enhance the dual-stream representation through causal cross-attention using the joint interaction state sequence of the live room as a conditional signal. After fusion, predict the future traffic peak.

2. The method for real-time traffic prediction in e-commerce live streaming based on multimodal learning according to claim 1, its features are as follows: The characteristic is that step S1 specifically includes: S11. Collect raw live data in units of a preset sliding time window. The raw live data includes live video stream and user behavior data. The user behavior data includes dynamic traffic characteristics and real-time interaction characteristics; S12. Perform frame extraction and audio separation on the live video stream, and extract feature representations of four modalities: text, visual, audio, and action; perform standardization processing on the user behavior data to generate a user behavior feature sequence; S13. Temporally align the modal representations and user behavior features using a Long Short-Term Memory (LSTM) network. For any target modality, treat the remaining modalities as a set of complementary modalities. Calculate the correlation between the target modality and each complementary modality using the Hadamard product. After softmax normalization, subtract the all-one vectors to obtain complementary weights. Weight the complementary modalities to obtain non-redundant compensation features. Summate these features to obtain the compensation representation of the target modality. Iterate through all time steps to obtain the compensation representation sequence for each modality. S14. For each target modality, its alignment representation is used as the query and the compensation representation is used as the key and value. After causal cross-attention interaction, it is added to the original alignment representation and then causal self-attention aggregation to obtain the final fused representation of the modality. By splicing together the final fusion representations of all modalities, the spatiotemporal multimodal representation of the broadcaster is obtained; S15. The user behavior feature sequence is passed through a long short-term memory network to obtain a multivariate temporal representation of the audience.

3. The method for real-time traffic prediction in e-commerce live streaming based on multimodal learning according to claim 1, characterized in that, Step S2 specifically includes: S21. For the anchor representation sequence, obtain the anchor-side historical representation through causal self-attention, and obtain the anchor-side interactive representation through causal cross-attention with the anchor representation as query and the audience representation as key and value; the causal cross-attention refers to using a causal mask matrix to shield the information of future time steps in the attention calculation, so that the attention calculation of the current time step only depends on the information of the current and historical time steps. S22. For the audience representation sequence, obtain the audience-side historical representation through causal self-attention, and obtain the audience-side interactive representation through causal cross-attention with the audience representation as the query and the anchor representation as the key and value. S23. Adaptively weightedly fuse the broadcaster's historical representation and interaction representation at each time step using sigmoid gating to obtain the updated broadcaster's state representation; perform the same operation on the viewer's side to obtain the updated viewer's state representation. S24. The updated state representations of the broadcaster and the audience at each time step are spliced ​​together to obtain the joint interactive state sequence of the live broadcast room.

4. The method for real-time traffic prediction in e-commerce live streaming based on multimodal learning according to claim 1, characterized in that, Step S3 specifically includes: S31. Decouple the inbound flow characteristics and outbound flow characteristics, and input them into two independent long short-term memory networks for time series modeling to obtain inbound representation sequences and outbound representation sequences. S32. Using the joint interaction state sequence of the live room as the key and value, and using the entry representation sequence and the exit representation sequence as the query respectively, the two representation sequences are enhanced by causal cross-attention. S33. The enhanced entry and exit characterization sequences are spliced ​​together, fused through a feedforward layer, and then output through a regression layer as the predicted peak traffic value within a preset time period in the future. The peak traffic value is the highest instantaneous number of online users within the preset prediction window.

5. The method for real-time traffic prediction in e-commerce live streaming based on multimodal learning according to claim 2, characterized in that, Frame extraction of the live video stream is performed using multi-rate sampling. A first sampling rate is used to extract global visual frames to capture the overall picture information of the live room, and a second sampling rate higher than the first sampling rate is used to extract action frames to capture changes in the anchor's body movements. The first sampling rate is 1~2 frames / second, and the second sampling rate is 8~15 frames / second. The action frames are identified and extracted based on the pixel difference between adjacent frames exceeding a preset threshold, where the preset threshold is the root mean square of the difference in grayscale values ​​of corresponding pixels in adjacent frames.

6. The method for real-time traffic prediction in e-commerce live streaming based on multimodal learning according to claim 2, characterized in that, The original live data is sampled in units of a preset sliding time window, and the preset time window, time step and prediction window are preset values; the time window is 2.5 to 3.5 minutes, each time step is 3 to 6 seconds, and the prediction window is 0.5 to 1.5 minutes.

7. The method for real-time traffic prediction in e-commerce live streaming based on multimodal learning according to claim 2, characterized in that, The pre-trained networks corresponding to the four modalities include: Vision Transformer for the visual modality, S3D for the action modality, Librosa for extracting MFCCs features for the audio modality, and BERT for the language modality.

8. A real-time traffic prediction system for e-commerce live streaming based on multimodal learning, characterized in that, The system includes: The representation learning module is used to collect raw live data, extract multimodal representations of the anchor and temporal representations of the audience, and generate spatiotemporal multimodal representations of the anchor and multivariate temporal representations of the audience through complementary attention fusion. The main interaction module is used to obtain the joint interaction state sequence of the live broadcast room based on the host's spatiotemporal multimodal representation and the audience's multivariate temporal representation, through causal masking across attention and adaptive gating; The flow decoupling module is used to decouple the inbound and outbound flows into two streams, perform time-series modeling on them separately, and generate inbound and outbound characterization sequences. The traffic prediction module is used to predict future traffic peaks by using the joint interaction state sequence of the live room as a conditional signal, enhancing the entry and exit representation sequences through causal cross-attention respectively, and then fusing them.

9. An electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method for real-time traffic prediction of e-commerce live streaming based on multimodal learning as described in any one of claims 1-7.

10. A computer-readable storage medium having stored thereon computer program instructions, which, when executed by a processor, implement the method for real-time traffic prediction of e-commerce live streaming based on multimodal learning as described in any one of claims 1-7.

Images