Method and system for real-time conversion rate prediction of clothing live streaming based on multi-modal deep learning

Abstract

The application discloses a clothes live broadcast real-time conversion rate prediction method and system based on multi-modal deep learning, relates to the field of deep learning, and comprises the following steps: collecting multi-scene samples based on influence factors of live broadcast conversion rate and constructing sample clusters; performing multi-modal analysis according to the corresponding relationship between the sample clusters and the influence factors, determining the causal probabilistic relationship between the change of each mode data and the real-time conversion rate, and constructing a causal probabilistic network; performing segmentation annotation, and assembling multi-modal training and test data sets; training and converging the multi-modal encoder and the time sequence fusion predictor; and performing correlation deployment, wherein the multi-modal encoder is used for multi-modal data segmentation and encoding of a live broadcast picture, and the time sequence fusion predictor is used for real-time conversion rate prediction by using the segmented and encoded multi-modal data. The application solves the technical problem of insufficient prediction accuracy of the existing clothes live broadcast real-time conversion rate prediction, and achieves the technical effect of accurate prediction of the clothes live broadcast real-time conversion rate.

Description

Technical Field

[0001] This application relates to the field of deep learning, and in particular to a method and system for predicting real-time conversion rates in live-streaming apparel using multimodal deep learning. Background Technology

[0002] In the apparel live-streaming e-commerce sector, real-time conversion rate prediction is crucial for optimizing live-streaming strategies and boosting sales performance. Accurate predictions help merchants adjust live-streaming content and marketing methods in a timely manner to maximize profits. Currently, the main approach to addressing this issue is to build traditional prediction models based on a single data source (such as live-streaming sales data or simple user behavior data) to predict conversion rates. However, because these methods rely on only a single data source, they cannot comprehensively capture the complex impact of multi-dimensional and multi-modal information (such as live-streaming visuals, host language, and audience interaction) on conversion rates during the live-streaming process. This results in insufficient prediction accuracy and makes it difficult to meet the demand for real-time, precise predictions.

[0003] Currently, the technology for predicting real-time conversion rates in live-streaming apparel sales suffers from insufficient accuracy. Summary of the Invention

[0004] This application provides a method and system for predicting real-time conversion rates in live-streaming apparel using multimodal deep learning. It employs techniques such as analyzing factors influencing live-streaming conversion rates, collecting samples from multiple scenarios, building sample clusters, analyzing the correspondence between sample clusters and influencing factors, clarifying the causal probability relationship between changes in each modality of data and the real-time conversion rate, constructing a causal probability network, performing segmentation and labeling based on the causal probability network, forming multimodal training and testing datasets, building a model framework of a multimodal encoder and a temporal fusion predictor, training the model using the training and testing sets until convergence, obtaining a trained model, and deploying the multimodal encoder and temporal fusion predictor in conjunction. The encoder segments and encodes the multimodal data of the live-streaming footage, and the predictor uses the encoded data to predict the real-time conversion rate and output the results. These techniques solve the technical problem of insufficient prediction accuracy in existing live-streaming apparel conversion rate prediction methods, achieving a high degree of accuracy in predicting real-time conversion rates for live-streaming apparel.

[0005] This application provides a multimodal deep learning method for predicting real-time conversion rates in live-streaming apparel sales, comprising: analyzing the influencing factors of live-streaming conversion rates, and collecting samples from multiple scenarios based on the influencing factors to construct a sample cluster; performing multimodal analysis of the influencing factors according to the correspondence between the sample cluster and the influencing factors, determining the causal probability relationship between changes in each modality data and the real-time conversion rate, and constructing a causal probability network; performing segmentation and labeling based on the causal probability network to construct a multimodal training dataset and a test dataset; building a model framework for a multimodal encoder and a temporal fusion predictor, and training and converging the multimodal encoder and the temporal fusion predictor using the multimodal training dataset and the test dataset to obtain the multimodal encoder and the temporal fusion predictor; and deploying the multimodal encoder and the temporal fusion predictor in association, wherein the multimodal encoder is used to segment and encode multimodal data of the live-streaming image, and the temporal fusion predictor uses the segmented and encoded multimodal data to predict the conversion rate in real time, and outputs the real-time conversion rate prediction result of the live-streaming sales.

[0006] In a possible implementation, the factors influencing live stream conversion rates are analyzed, and the following processing is performed: data is classified according to the type of live stream data source coverage, including video streams, audio streams, user behavior, product context, and environmental factors; each data source type is decomposed into multiple granularities to obtain the data source granularity level; family-specific live stream data is extracted, including different conversion rate gradients; based on the family-specific live stream data, influencing factors are aligned according to the data source granularity level to fit the influence relationship of conversion rate fluctuations; the influence relationship of conversion rate fluctuations is filtered according to a preset fluctuation threshold to obtain the factors influencing the live stream conversion rate.

[0007] In a possible implementation, to obtain the influencing factors of the live streaming conversion rate, the following processing is also performed: a multi-level influencing factor dictionary is constructed according to the scene layer, modality layer, and fine-grained element layer. The fine-grained elements include product salience, posture type, and color / texture features in the video modality; volume, speech rate, and emotional fluctuations in the audio modality; discount terms and scarce cue words in the text modality; interaction rate and month-on-month increase in add-to-cart in the behavioral modality; and price range, discount intensity, and inventory status in the product and context modality. Based on the hierarchical feature tags in the multi-level influencing factor dictionary, the influencing factors of the live streaming conversion rate are projected in multiple levels to form a multi-level dictionary structure of live streaming conversion rate influencing factors.

[0008] In possible implementations, a sample cluster is constructed, and the following processing is performed: stratified sampling is carried out in multiple scenarios based on a multi-level influencing factor dictionary, including multi-scale samples with micro-time windows, medium-time windows, and large-time windows; oversampling is performed on scarce composite events based on multi-scale samples, and supplementary samples with similar scenarios are introduced for new anchors or new products to construct a multi-scenario sample cluster.

[0009] In a possible implementation, a causal probability network is constructed, and the following processing is performed: Based on the correspondence between sample clusters and influencing factors, a multimodal causal probability network is established using a time-series causal structure learning method. Nodes include various fine-grained elements, and edges include time dependencies and intermodal causal relationships. Instrumental variables and proximal causal modeling are introduced. Based on the multimodal causal probability network, the local average processing effect of each fine-grained element on the real-time conversion rate is estimated to form a causal probability network capable of performing do-operations, which is used to derive the causal probability relationship between multimodal data changes and the conversion rate.

[0010] In a possible implementation, segmentation annotation is performed based on the causal probability network to construct a multimodal training dataset and a test dataset. The following processing is then performed: weakly supervised segmentation annotation is performed on the multimodal data based on video object detection, audio event detection, and text keyword matching, and the segmentation results are mapped to nodes of the causal probability network to construct causal relationships in the segmented data; based on the causal relationships in the segmented data, mapping and annotation are performed with the sample conversion rate relationship to construct a sample dataset, and the training samples and test samples are divided according to a preset ratio, and sample time-series labels are established according to the temporal relationship of multi-scale samples to obtain the training dataset and the test dataset.

[0011] In a possible implementation, weakly supervised segmentation and labeling of multimodal data is performed based on video object detection, audio event detection, and text keyword matching. The segmentation results are then mapped to causal probability network nodes to construct causal relationships for the segmented data. The following processing is then performed: Based on the influence relationship between video object detection and conversion rate, the live stream is segmented to identify salient product areas, human postures, and display actions; based on the influence relationship between audio event detection and conversion rate, the live stream audio is segmented to identify speech rate, volume changes, and sound effects; based on the influence relationship between text keyword matching and conversion rate, the live stream speech recognition results and bullet screen text are segmented to identify discount statements, scarce prompts, and purchase intent phrases; based on the mapping of the segmentation results to the causal probability network node states, the influence relationship labels between segmented data features and conversion rate are determined, and the segmented data is labeled with causal relationships to obtain the causal relationships of the segmented data.

[0012] In a possible implementation, based on the causal relationship of the segmented data, a mapping and labeling with the sample conversion rate relationship is performed to construct a sample dataset. Training and test samples are divided according to a preset ratio, and temporal labels are established based on the temporal relationship of multi-scale samples to obtain the training and test datasets. The following processing is then performed: Causal relationship labels of segmented segments are mapped to conversion rate labels of corresponding time windows to form causal-conversion aligned samples; the aligned samples are divided into training and test sets according to a preset ratio, avoiding simultaneous appearance of samples from the same broadcaster or the same scene in both sets; the training and test sets are organized according to micro, medium, and large time windows to establish a multi-scale temporal sample set; temporal labels are generated for continuous windows within each scale to support multi-scale temporal fusion modeling.

[0013] In a possible implementation, the multimodal encoder and temporal fusion predictor are trained and converged using the multimodal training dataset and test dataset, and the following processing is performed: Training batches are constructed based on causal-conversion aligned samples, with the original segmented fragments used as anchor samples, and high-conversion fragments corresponding to the anchor samples or in the same scenario are selected as positive samples; counterfactual negative samples are generated by locally replacing controllable elements according to the causal probability network, where the controllable elements include at least one of discount intensity, product salience, frequency of rare prompt words, and peak speech rate, while keeping other modal features unchanged; during model training, the prediction differences between anchor samples and positive and negative samples are compared, and by minimizing the difference between anchor samples and positive samples and widening the difference between positive and negative samples, the model gradually learns to distinguish between high-quality and low-quality operational scenarios; training and validation are performed according to three time scales: micro-time window, medium-time window, and large-time window, and model parameters are adjusted based on test and validation results during training, while the output prediction results are calibrated to ensure consistency between the predicted values ​​and the actual conversion rates, thus achieving convergence.

[0014] This application also provides a multimodal deep learning-based real-time conversion rate prediction system for live-streaming apparel, comprising: a multi-scenario sample collection module for analyzing the influencing factors of live-streaming conversion rate and collecting multi-scenario samples based on the influencing factors to construct a sample cluster; a causal probability network construction module for performing multimodal analysis of influencing factors based on the correspondence between the sample cluster and the influencing factors, determining the causal probability relationship between changes in each modality data and the real-time conversion rate, and constructing a causal probability network; a segmentation and labeling module for performing segmentation and labeling based on the causal probability network to construct a multimodal training dataset and a test dataset; a training module for building a model framework for a multimodal encoder and a temporal fusion predictor, and training and converging the multimodal encoder and the temporal fusion predictor using the multimodal training dataset and the test dataset to obtain the multimodal encoder and the temporal fusion predictor; and an association and deployment module for associating and deploying the multimodal encoder and the temporal fusion predictor, wherein the multimodal encoder is used to perform multimodal data segmentation and encoding on the live-streaming image, and the temporal fusion predictor uses the segmented and encoded multimodal data to perform real-time conversion rate prediction and output the real-time conversion rate prediction result for the live stream.

[0015] This application proposes a method and system for predicting real-time conversion rates in live-streaming apparel using multimodal deep learning. First, it analyzes the influencing factors of live-streaming conversion rates and collects samples from multiple scenarios based on these factors to construct a sample cluster. Then, based on the correspondence between the sample cluster and the influencing factors, it performs multimodal analysis of the influencing factors to determine the causal probability relationship between changes in each modality's data and the real-time conversion rate, constructing a causal probability network. Next, it performs segmentation and labeling based on the causal probability network to build multimodal training and testing datasets. Then, it constructs a model framework for a multimodal encoder and a temporal fusion predictor. The multimodal encoder and temporal fusion predictor are trained and converged using the multimodal training and testing datasets to obtain the multimodal encoder and temporal fusion predictor. Finally, the multimodal encoder and temporal fusion predictor are deployed in a correlated manner. The multimodal encoder is used to segment and encode multimodal data from the live-streaming footage, and the temporal fusion predictor uses the segmented and encoded multimodal data to predict the conversion rate in real time, outputting the real-time conversion rate prediction result. This achieves the technical effect of accurately predicting the real-time conversion rate of live-streaming apparel. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments of the present invention will be briefly described below. Flowcharts are used in this application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed precisely in sequence. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from these processes.

[0017] Figure 1 This is a flowchart illustrating the multimodal deep learning-based real-time conversion rate prediction method for live-streaming clothing provided in this application embodiment.

[0018] Figure 2 This is a schematic diagram of the structure of a multimodal deep learning-based real-time conversion rate prediction system for live-streaming clothing, provided in an embodiment of this application.

[0019] Figure labeling: Multi-scene sample acquisition module 10, Causal probability network construction module 20, Segmentation and labeling module 30, Training module 40, Association and deployment module 50. Detailed Implementation

[0020] The above description is merely an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below.

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description of this application will be provided in conjunction with the accompanying drawings. The described embodiments should not be considered as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] In the following description, references to "some embodiments" describe a subset of all possible embodiments; however, it is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. The terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only.

[0023] This application provides a multimodal deep learning-based method for predicting real-time conversion rates in live-streaming apparel sales, such as... Figure 1 As shown, the method includes:

[0024] Step S100: Analyze the factors affecting the live streaming conversion rate, and collect samples from multiple scenarios based on the factors to build a sample cluster.

[0025] Specifically, identify the factors influencing livestream conversion rates, such as clothing display effects (images), host explanations (audio), chat interaction (text), product discount information (text), and livestream duration (time series). Design multi-scenario sample collection schemes, such as collecting data from livestreams of different clothing brands (sports, casual, formal, etc.), different livestream times (daytime, evening), and different host styles (humorous, professional, friendly). Web scraping technology can be used to collect publicly available livestream data from livestream platforms, including video streams, chat comments, likes, and purchase data. For example, a web scraping program can be written using the Scrapy framework to periodically crawl livestream data corresponding to the URLs of clothing livestreams and store it in a local database. Alternatively, cameras, microphones, and other sensor devices can be deployed in the livestream scene to collect livestream images and sound. For example, a high-definition camera can be used to capture livestream video images at a specific frame rate, and a microphone can be used to capture audio signals at a certain sampling rate. Store the collected modal data (images, audio, text, time series) according to a unified sample format to build a sample cluster. For example, each sample contains a video file of the live stream segment, the corresponding speech-to-text text, bullet screen text, product information text, time series data (such as the number of viewers per minute and the number of likes), and the corresponding conversion rate tags (such as the purchase amount and the number of buyers).

[0026] For example, in a live stream for a sportswear brand, a high-definition camera was used to capture the live stream footage at a frame rate of 30 frames per second and saved as an MP4 video file; simultaneously, a microphone was used to capture the host's audio at a sampling rate of 44.1kHz and saved as a WAV audio file; in addition, web crawlers were used to capture viewers' comments during the live stream and stored as JSON text files; the e-commerce platform system recorded time-series data such as the number of viewers and likes per minute for the live stream and saved it in CSV format; finally, the purchase conversion rate of the live stream was used as a sample label.

[0027] In one possible implementation, analyzing the factors influencing live stream conversion rates, step S100 further includes step S110, classifying data according to the types of live stream data sources, including video streams, audio streams, user behavior, product context, and environmental factors. Specifically, the collected live stream data is classified, and data tagging technology is used to label data sources such as video streams, audio streams, user behavior, product context, and environmental factors. For example, at the live stream platform's data interface, the data is divided into different categories based on the data type information returned by the interface. Video stream data is stored in formats such as MP4, audio stream data is stored in formats such as WAV, user behavior data records behavior types (such as clicks, dwell times, purchases), behavior timestamps, etc., in formats such as JSON, product context data including product name, price, size, color, etc., is stored in a structured table format, and environmental factor data such as live stream time and location are recorded in text or a specific encoding format.

[0028] For example, in a live clothing broadcast, the collected data can be categorized as follows: video stream data consists of the video files of the live broadcast; audio stream data consists of the audio files of the host's explanations; user behavior data records viewers' clicks, purchases, and dwell time in different clothing display segments; product context data includes detailed information such as the brand, style, material, and price of the clothing; and environmental factor data includes information such as the time of the live broadcast (e.g., 8 pm) and the location of the live broadcast (e.g., an indoor live broadcast room).

[0029] Step S120 involves multi-granularity decomposition of each data source type to obtain the data source granularity hierarchy. Specifically, hierarchical clustering algorithms are used to decompose each data source into granularities. Taking video streams as an example, the video can first be divided into multiple segments based on time (e.g., one segment every 10 seconds), which is coarse-grained decomposition. Then, each segment is further subdivided according to the clothing display parts in the frame (e.g., tops, pants) and display actions (e.g., rotating display, model trying on), to obtain fine-grained data. For user behavior data, coarse-grained segmentation can be performed based on behavior type (e.g., browsing, clicking, purchasing), and then fine-grained segmentation can be performed based on the specific product in which the behavior occurred (e.g., the specific style of a garment).

[0030] Step S130: Extract live stream data from the same category, including different conversion rate gradients. Specifically, using clustering algorithms in data mining (such as K-Means), live stream data with similar characteristics are clustered into the same category. For example, live stream data can be clustered based on features such as clothing type (e.g., all live streams are sportswear), streamer style (e.g., all streamers are humorous), and live stream time (e.g., all live streams are in the evening). Simultaneously, the conversion rate of each live stream is calculated and divided according to conversion rate (e.g., low gradient: conversion rate <10%, medium gradient: 10% ≤ conversion rate <30%, high gradient: conversion rate ≥30%), extracting live stream data from the same category containing different conversion rate gradients.

[0031] Step S140: Based on the same-family live streaming data, align the influencing factors according to the granularity level of the data sources to fit the relationship between conversion rate fluctuations. Specifically, align the features of each data source in the same-family live streaming data according to the granularity level. For example, for video streams and user behavior data, align them along the time dimension, corresponding the video display content (such as a display segment of a certain garment) within the same time period with user behaviors (such as clicks and purchase operations) within that time period. For product context data and environmental factor data, align them with video streams and user behavior data, such as associating the detailed information of a certain garment with the video segment displaying the garment and the corresponding user behavior data. Use regression analysis or time series analysis methods to fit the relationship between conversion rate fluctuations and various influencing factors. For example, using clothing display characteristics in the video stream (such as display duration and display area), explanation characteristics in the audio stream (such as explanation duration and keyword frequency), user behavior characteristics (such as dwell time and number of clicks), product context characteristics (such as price and discount level), and environmental factors (such as live broadcast time) as independent variables, and conversion rate as the dependent variable, a regression model is established to fit the influence relationship of each factor on the fluctuation of conversion rate.

[0032] For example, for live streaming data of sportswear, the time period in the video stream showing the rotating action of the clothing top is aligned with the user's click behavior during that time period. At the same time, combined with price information in the product context and live streaming time in the environmental factors, a linear regression model is used to fit the relationship between these factors and the conversion rate. For example, it was found that for every 1 minute increase in the display time of the rotating clothing top, the conversion rate increases by 2%; for every 10% decrease in price, the conversion rate increases by 5%, and so on.

[0033] Step S150: The influence relationship of conversion rate fluctuations is filtered according to a preset fluctuation threshold to obtain the influencing factors of the live streaming conversion rate. Specifically, a threshold for conversion rate fluctuations is set, such as defining significant fluctuations as an absolute change exceeding 5% or a relative change exceeding 10%. Based on the fitted influence relationship of conversion rate fluctuations, those influencing factors whose impact on conversion rate fluctuations reaches the preset threshold are screened out. For example, statistical significance tests (such as t-tests and F-tests) are used to determine which factors contribute significantly to conversion rate fluctuations in the fitted relationship, i.e., their corresponding regression coefficients and other parameters are statistically significant, thereby screening out the factors that truly have a significant impact on the conversion rate.

[0034] For example, if the preset conversion rate fluctuation threshold is 5%, after screening, it is found that in the live streaming data of sportswear, the impact of the rotation display time of clothing tops on the conversion rate reaches this threshold, while the impact of the display time of clothing pants on the conversion rate does not reach the threshold. Therefore, it is determined that the rotation display time of clothing tops is one of the factors affecting the live streaming conversion rate, while the display time of clothing pants is not a key factor, thus obtaining the key factors affecting the live streaming conversion rate.

[0035] In one possible implementation, to obtain the influencing factors of the live stream conversion rate, step S100 further includes step S160, constructing a multi-level influencing factor dictionary according to the scene layer, modal layer, and fine-grained element layer. The fine-grained elements include product salience, posture type, and color / texture features in the video modality; volume, speech rate, and emotional fluctuations in the audio modality; discount terms and scarce cue words in the text modality; interaction rate and month-on-month increase in add-to-cart in the behavioral modality; and price range, discount intensity, and inventory status in the product and context modality. Specifically, the structure of the multi-level influencing factor dictionary is designed using methods such as the Analytic Hierarchy Process (AHP). The influencing factors are refined layer by layer, from the macro-level scene layer to the specific fine-grained element layer. For example, first determine the broad category of the live streaming scenario (e.g., clothing live streaming), which is the scenario layer; then divide it into modal layers such as video, audio, text, behavior, and product and context; finally, further subdivide each modal layer into specific fine-grained elements, such as product salience (salience, average, insignificance), posture type (static display, model trying on, rotating display), and color / texture features (bright colors, dark colors, simple textures, complex textures) in the video modal; volume (high, medium, low), speech rate (fast, normal, slow), and emotional fluctuation (positive, negative, neutral) in the audio modal; discount terms (spend more, discount, buy one get one free), and scarcity keywords (limited quantity, flash sale, only left) in the text modal; interaction rate (high, medium, low, measured by the number of interactions per unit time) and month-on-month increase in add-to-cart (positive, negative, compared to the previous period) in the behavior modal; and price range (high, mid-range, low), discount intensity (high discount, low discount), and inventory status (sufficient, tight, out of stock) in the product and context modal. Data annotation tools (such as LabelImg) and natural language processing technologies (such as part-of-speech tagging and entity recognition) are used to label and classify the factors influencing live stream conversion rates. For example, image annotation tools are used to label the salience of products in videos (such as salience, average, insignificance), speech-to-text software combined with sentiment analysis algorithms is used to label the emotional fluctuations in audio (such as positive, negative, neutral), and text analysis tools are used to identify and label discount statements and scarce keywords in text.

[0036] Step S170: Based on the hierarchical feature labels in the multi-level influencing factor dictionary, the influencing factors of the live streaming conversion rate are projected in multiple levels to form a multi-level dictionary structure of live streaming conversion rate influencing factors. Specifically, a data mapping algorithm is used to match and project the extracted influencing factors with the hierarchical feature labels in the multi-level influencing factor dictionary. For example, based on a text matching algorithm, the discount information (such as "spend 200 and get 50 off") extracted from the text is mapped to the discount label (spend 200 and get 50 off) under the text modality; based on an image feature matching algorithm, the bright color features of the products in the video are mapped to the color feature label (bright color) under the video modality. A relational database (such as MySQL) or a non-relational database (such as MongoDB) is used to store the multi-level dictionary structure of live streaming conversion rate influencing factors. For example, in a relational database, scene layer tables, modality layer tables, and fine-grained feature layer tables are created, and the data at each level is linked through foreign keys to form a hierarchical data structure; in a non-relational database, the data at each level is stored in the form of nested documents.

[0037] In one possible implementation, a sample cluster is constructed, and step S100 further includes step S180, which involves stratified sampling based on a multi-level influencing factor dictionary across multiple scenarios, including multi-scale samples with micro-time windows, medium-time windows, and large-time windows. Specifically, based on the characteristics of the live streaming scenario and the multi-level influencing factor dictionary, stratified sampling methods are used to collect samples at different time scales. For example, the live stream is stratified according to different stages (opening, clothing display, interactive segment, and ending), and samples are collected for micro-time windows (e.g., 1-5 minutes), medium-time windows (e.g., 10-30 minutes), and large-time windows (e.g., 1-2 hours) at each stage. Time series analysis tools (e.g., the Pandas library) are used to divide the time windows, and data collection tools are combined with these tools for data collection. For example, the Pandas rolling or resample functions are used to divide the time windows, and multimodal data such as video, audio, text, and behavior within the corresponding time windows are collected through web crawling, database queries, and other means.

[0038] Step S190 involves oversampling scarce composite events based on multi-scale samples, and introducing supplementary samples of similar scenarios for new anchors or new products to construct a multi-scenario sample cluster. Specifically, data mining algorithms (such as association rule mining and anomaly detection algorithms) are used to identify scarce composite events, such as the low-frequency situation of "a new anchor promoting a high-priced new product for the first time and experiencing a live broadcast accident but with an abnormally high conversion rate." The samples corresponding to these events are oversampled using algorithms such as SMOTE (Synthetic Minority Oversampling) and ADASYN (Adaptive Synthetic Sampling Method) to generate synthetic samples, increasing their quantity in the sample cluster and enabling the model to learn the characteristics of these special cases.

[0039] Similarity measurement algorithms (such as cosine similarity and Euclidean distance) are used to find similar scenario samples to supplement new anchors or new products. For example, for new anchors, the similarity between them and existing anchors in terms of style (such as speaking speed and explanation style), live broadcast time, target audience, etc., is analyzed, and samples corresponding to anchors with high similarity are selected as supplements; for new products, live broadcast samples of existing similar products are matched according to the characteristics of the new product such as type, price range, target audience, etc.

[0040] Use data warehouse tools (such as Hive) or distributed file systems (such as HDFS) to store and manage multi-scenario sample clusters. Categorize, index, and version control the samples for efficient data retrieval and updates.

[0041] Step S200: Based on the correspondence between the sample cluster and the influencing factors, perform multimodal analysis of the influencing factors, determine the causal probability relationship between the changes in each modality data and the real-time conversion rate, and construct a causal probability network.

[0042] Specifically, for each sample in the sample cluster, features of each modality are extracted. For example, for image modality, features such as the proportion of clothing in the image, display angle, and color features are extracted; for speech modality, features such as speech rate, tone, and frequency of keyword occurrence are extracted; for text modality (bullet comments, product descriptions, etc.), features such as word frequency, sentiment, and topic classification are extracted; and for time series modality, features such as the trend of changing number of viewers and changes in interaction frequency are extracted.

[0043] Based on the extracted features, causal inference methods from statistics and machine learning, such as structural causal models (SCM) and Bayesian networks, are used to analyze the causal relationships between various modalities and conversion rates, constructing a causal probability network. For example, image features, speech features, text features, and time series features are used as causal variables, and conversion rate is used as the outcome variable. Bayesian network learning algorithms (such as the EM algorithm and MCMC algorithm) are used to estimate the conditional probability distribution of each causal variable with respect to the outcome variable, determining the causal probability relationship between changes in each modality of data and the real-time conversion rate, and constructing a causal probability graphical model (causal probability network). In the graph, nodes represent each feature and conversion rate, and edges represent the causal relationships between them and their corresponding probability values.

[0044] For example, when analyzing the relationship between clothing display effect (image modality) and conversion rate, it was found that when the proportion of clothing in the image exceeds a certain threshold (e.g., 60%) and the display angle is 360° all-around, the causal probability of this combination of image features improving the conversion rate is 0.75 in the causal probability network, meaning that the probability of improving the conversion rate is 75% when these image features are met. When analyzing the content of the anchor's explanation (voice modality), it was found that when the speaking speed is moderate (150-180 words per minute) and contains specific keywords (e.g., "limited-time discount" "new product launch"), the corresponding causal probability is 0.68, indicating that the probability of a high conversion rate under this voice feature is 68%.

[0045] In one possible implementation, a causal probability network is constructed. Step S200 further includes step S210, which establishes a multimodal causal probability network using a temporal causal structure learning method based on the correspondence between sample clusters and influencing factors. Nodes include fine-grained elements, and edges include temporal dependencies and intermodal causal relationships. Specifically, temporal causal structure learning methods such as Granger causality analysis and dynamic Bayesian networks are used to analyze the temporal and causal relationships between fine-grained elements in the sample cluster. For example, Granger causality tests are used to determine whether significant changes in product characteristics in the video modality have a causal impact on the interaction rate in the subsequent user behavior modality. A multimodal causal probability network is constructed using graph neural networks (GNNs), with fine-grained elements as nodes and temporal dependencies and intermodal causal relationships as edges. For example, the message passing mechanism of GNNs is used to propagate and update feature information between nodes, learning complex causal relationships between nodes.

[0046] For example, in a sample cluster of clothing live streams, the multimodal causal probability network is constructed as follows: Nodes include product salience, posture type, and color / texture features in the video modality; volume, speech rate, and emotional fluctuations in the audio modality; discount information and scarce cue words in the text modality; interaction rate and incremental addition to cart in the behavioral modality; and fine-grained elements such as price range, discount strength, and inventory status in the product and context modality. For instance, there exists a directed edge from product salience in the video modality to interaction rate in the behavioral modality, indicating that changes in product salience affect the audience's interaction rate; there also exists a directed edge from emotional fluctuations in the audio modality to incremental addition to cart in the behavioral modality, indicating that changes in the broadcaster's emotional fluctuations affect the audience's cart-adding behavior.

[0047] Step S220 introduces instrumental variables and proximal causal modeling. Based on the multimodal causal probability network, it estimates the local average processing effect of each fine-grained element on the real-time conversion rate, forming a causal probability network capable of do-operations, used to deduce the causal probability relationship between multimodal data changes and the conversion rate. Specifically, instrumental variables are selected, such as live broadcast time and streamer popularity. These variables are not affected by other fine-grained elements but are related to the real-time conversion rate. Instrumental variable methods, such as two-stage least squares (2SLS), are used to estimate the causal effect of each fine-grained element on the real-time conversion rate. For example, using the live broadcast time as an instrumental variable, the local average processing effect of pose type changes in the video modality on the conversion rate is estimated. Proximal causal models, such as causal forests and dual machine learning methods, are employed to handle complex causal relationships and high-dimensional data. For example, using the causal forest algorithm, based on multimodal data in the sample cluster, the local average processing effect of speech rate changes in the audio modality on the conversion rate is estimated, while controlling for interference from other variables. By constructing a causal probability network capable of do-operations, we can simulate the real-time changes in conversion rates when intervening in a fine-grained element. For example, we can use do-operations to estimate the expected change in conversion rate when the discount description in a text modality is changed from "spend more to save more" to "discount".

[0048] For example, assuming we choose the live streaming time slot (e.g., prime time, non-prime time) as the instrumental variable, it is related to the real-time conversion rate (prime time usually has a higher conversion rate) but is not directly affected by other fine-grained factors. Using a two-stage least squares method, we first estimate the relationship between the live streaming time slot and the pose type in the video modality, and then estimate the impact of pose type on the conversion rate, thus obtaining the local average treatment effect of pose type changes on the conversion rate. For example, we find that when the pose type changes from "static display" to "model trying on," the conversion rate increases by an average of 10%. Using a causal forest algorithm, with speech rate and emotional fluctuations in the audio modality as features and real-time conversion rate as the label, we train the model to estimate the local average treatment effect of speech rate changes on the conversion rate. Assuming the model estimation results show that when the speech rate changes from "normal" to "fast," the conversion rate increases by an average of 5%. In the constructed causal probability network, we perform a do-operation simulation, setting the discount caliber in the text modality to "discount," i.e., do(discount caliber = "discount"), and observe the changes in the real-time conversion rate. Assuming the results show that setting it to "discount" will increase the conversion rate by 8% compared to keeping the original discount term "spend more than enough".

[0049] Step S300: Perform segmentation and labeling based on the causal probability network to construct a multimodal training dataset and a test dataset.

[0050] Specifically, based on the causal relationships and probability values ​​in the causal probability network, data labeling software, such as LabelImg and LabelStudio, is used to segment and label the sample data. For example, for image samples, based on the causal probability corresponding to the clothing display effect, the image samples are labeled as "high conversion rate related images" or "low conversion rate related images"; for audio samples, based on the causal probability corresponding to the explanation content, they are labeled as "conversion-promoting audio" or "normal audio"; for text samples (bullet comments, product introductions, etc.), based on the causal probability between text features and conversion rate, they are labeled as "positive review text," "neutral review text," "negative review text," or "marketing text," "non-marketing text," etc.

[0051] According to data segmentation techniques, labeled sample data is divided into training and testing datasets according to certain rules and proportions. For example, random sampling can be used to divide the dataset into training and testing sets in an 8:2 ratio; alternatively, based on time series characteristics, earlier data can be used as the training set and later data as the testing set to simulate actual spatiotemporal changes. Data augmentation operations are then performed on the training dataset to generate the final multimodal training and testing datasets. For example, image data in the training set can be augmented by horizontal flipping or rotation by a certain angle (e.g., ±15°) to increase the number of image samples; speech data can be augmented by adding white noise or changing the speech rate by ±10%; and text data can be augmented by synonym replacement (replacing a certain proportion of words) to generate more training samples and improve the model's adaptability to different data variations.

[0052] In one possible implementation, segmentation and annotation are performed based on the causal probability network to construct a multimodal training dataset and a test dataset. Step S300 further includes step S310, which involves weakly supervised segmentation and annotation of the multimodal data based on video object detection, audio event detection, and text keyword matching, and mapping the segmentation results to nodes of the causal probability network to construct causal relationships in the segmented data. Specifically, object detection algorithms (such as Faster R-CNN) are used to segment and annotate the video modality. For example, in a live-streaming video of clothing, target objects such as the main body of the clothing, models, and display props in each frame are detected, and their positions and categories are labeled. Audio event detection models (such as CNN-based audio classification models) are used to identify key events in the audio modality. For example, events such as the host explaining clothing features, promotional activities, and interactive segments are identified. Text matching algorithms (such as BM25 and word vector-based similarity matching) are used to perform keyword matching and annotation of the text modality (such as bullet comments and product introductions). For example, keywords such as discount information ("discount", "full reduction") and product attributes ("material", "size") are matched.

[0053] The segmentation and annotation results of video, audio, and text are matched and mapped to nodes in a causal probability network. For example, a significant change in clothing detected in a video is mapped to a product salience node in the causal probability network, and a promotional explanation detected in an audio is mapped to an emotion fluctuation node in the audio modality.

[0054] Step S320: Based on the causal relationship of the segmented data, map and label it with the sample conversion rate relationship to construct a sample dataset. Divide the data into training and testing samples according to a preset ratio, and establish sample time-series labels according to the temporal relationship of multi-scale samples to obtain the training and testing datasets. Specifically, use an association rule mining algorithm (such as Apriori) to map and label the causal relationship of the segmented data with the sample conversion rate. For example, when the "highly significant clothing display" event and the "increased audience interaction rate" event occur simultaneously and the conversion rate increases, it is labeled as a valid causal relationship. Use data sampling methods (such as random sampling and stratified sampling) to construct the sample dataset and divide it according to a preset ratio (such as 80% training set and 20% test set). For example, use the `train_test_split` function of sklearn for partitioning. Based on the temporal relationship of multi-scale samples, use time-series labeling tools (such as Pandas' time-series functionality) to establish sample time-series labels. For example, label micro-time window samples with minute-level time-series labels and mid-time window samples with hour-level time-series labels.

[0055] In one possible implementation, weakly supervised segmentation and annotation of multimodal data is performed based on video object detection, audio event detection, and text keyword matching. The segmentation results are then mapped to nodes in a causal probability network to construct causal relationships among the segmented data. Step S310 further includes step S311, which segments the live stream frame based on the influence relationship between video object detection and conversion rate, identifying salient product regions, human poses, and display actions. Specifically, object detection models (such as Faster R-CNN or YOLO) are used to segment the live stream frame and identify salient product regions. The saliency index of the product in the frame is calculated. If the saliency is greater than a preset threshold, it is mapped to the "product saliency" node state in the causal probability network. Skeleton keypoint detection technology (such as OpenPose) is used to identify human poses and display actions in the frame, such as rotational display actions. These action features are converted into state labels and mapped to corresponding nodes in the causal probability network.

[0056] Step S312 involves segmenting the live audio based on the relationship between audio event detection and conversion rate, identifying speech rate, volume changes, and sound effect events. Specifically, a speech rate threshold is dynamically calculated based on the number of syllables within a time window detected using a speech processing library (such as Librosa). When the speech rate exceeds the threshold, it is mapped to the state of the "speech rate peak" node in the causal probability network. Volume changes are identified using short-time energy calculations, and volume levels are mapped to nodes in the causal probability network. Sound effect events are identified using a pre-trained audio classification model (such as VGGish) and mapped to corresponding nodes.

[0057] Step S313: Based on the influence relationship between text keyword matching and conversion rate, the live broadcast voice recognition results and bullet screen text are segmented to identify discount information, scarce prompt words, and purchase intent phrases. Specifically, a scarce prompt word knowledge base (such as "only left" and "limited quantity") is constructed. Text vector similarity matching (based on pre-trained word vector models such as BERT) and semantic similarity algorithms (such as Sentence-BERT) are used to identify discount information, scarce prompt words, and purchase intent phrases from the live broadcast voice recognition results and bullet screen text. When the similarity between the identified result and the phrase in the knowledge base exceeds a threshold (such as 0.7), it is mapped to the "scarcity prompt" node state in the causal probability network.

[0058] Step S314: Based on the mapping of the segmentation results to the node states of the causal probability network, determine the label of the influence relationship between the segmented data features and the conversion rate, and identify the causal relationship of the segmented data to obtain the causal relationship of the segmented data. Specifically, after completing the feature extraction of video, audio, and text modalities, these segmentation results are mapped to the corresponding node states of the causal probability network according to preset mapping rules. For example, "product salient area" in the video modal corresponds to the "product salience" node, "high volume" in the audio modal corresponds to the "volume change" node, and "scarce cue words" in the text modal corresponds to the "scarce cue" node. By checking the consistency between the annotation results and the node states of the causal probability network, it is ensured that each annotation result can accurately reflect the influence relationship of the conversion rate, thereby identifying the causal relationship of the segmented data and forming the causal relationship of the segmented data.

[0059] In one possible implementation, based on the causal relationship of the segmented data, a mapping and labeling with the sample conversion rate relationship is performed to construct a sample dataset. Training and test samples are divided according to a preset ratio, and time-series labels are established based on the temporal relationship of multi-scale samples to obtain the training and test datasets. Step S320 further includes step S321, mapping the causal relationship labels of the segmented segments with the conversion rate labels of the corresponding time windows to form causal-conversion aligned samples. Specifically, for each segmented segment, its causal relationship label (e.g., "significant improvement in product effectiveness," "peak speech rate," etc.) and the conversion rate label of the corresponding time window (e.g., "conversion rate increase," "conversion rate decrease," etc.) are obtained. Using a data matching algorithm, the causal relationship label and conversion rate label are aligned according to timestamps. For example, in Pandas, the conversion rate data is set as the right table, and the causal relationship data as the left table. A merging operation is performed according to timestamps, allowing for approximate matching with a certain time difference (e.g., 5 seconds). According to label fusion rules, the aligned causal relationship label and conversion rate label are combined into a composite label to form a causal-conversion aligned sample.

[0060] Step S322: Divide the aligned samples into training and test sets according to a preset ratio, avoiding the simultaneous appearance of samples from the same broadcaster or scene in both sets. Specifically, extract the broadcaster ID and scene feature information from the aligned samples as the basis for sample grouping. Use a stratified sampling algorithm to divide the samples according to a preset ratio (e.g., 80% training set, 20% test set), setting the stratify parameter to broadcaster ID and scene features to ensure that the distribution of broadcasters and scenes in the training and test sets is consistent with the original data. Filter the samples in the divided training and test sets to check if samples from the same broadcaster or scene appear in both sets simultaneously. If so, adjust the distribution, assigning conflicting samples to the same set.

[0061] Step S323 involves organizing the training and test sets according to micro, medium, and large time windows to establish multi-scale time-series sample sets. Specifically, for each sample in the training and test sets, its timestamp information is obtained. Time window partitioning techniques are used to divide the samples according to the duration of micro, medium, and large time windows. For example, the Pandas resample function is used to group the samples according to time intervals of 1 minute, 5 minutes, and 10 minutes. The partitioned samples are then reorganized according to the order of the time windows to form micro, medium, and large time window sets. For example, samples within a 1-minute time window form a micro time window sample set, samples within a 5-minute time window form a medium time window sample set, and samples within a 10-minute time window form a large time window sample set.

[0062] Step S324 involves generating temporal labels for continuous windows within each scale to support multi-scale temporal fusion modeling. Specifically, for the temporal sample set at each scale (micro-time window, medium-time window, large-time window), temporal labels are generated according to the order of the samples within the time window. For example, in the micro-time window set, the temporal label of the first sample is set to 0, the second sample to 1, and so on. Label encoding techniques are used to convert the temporal labels into a model-recognizable format. For example, for the temporal labels 0, 1, 2…n in the micro-time window set, one-hot encoding is used to convert them into a vector representation of dimension n+1; or embedding encoding is used to map the temporal labels to a low-dimensional vector space.

[0063] Step S400: Build the model framework of the multimodal encoder and the temporal fusion predictor, and use the multimodal training dataset and the test dataset to train and converge the multimodal encoder and the temporal fusion predictor to obtain the multimodal encoder and the temporal fusion predictor.

[0064] Specifically, a multimodal encoder is a model component used to encode features from data of different modalities. It converts raw multimodal data into fixed-length feature vectors, extracting effective information from the data for subsequent fusion and prediction. A time-series fusion predictor is a fusion and prediction model for time-series data. It can fuse multimodal features while considering time-series characteristics and predict future time-series metrics (such as real-time conversion rates) based on historical data.

[0065] Multimodal encoders are built using deep learning frameworks such as TensorFlow and PyTorch to encode modal data including images, speech, text, and time series data. For example, a convolutional neural network (CNN) encoder is built to process image data, with the input image passing through convolutional and pooling layers to extract feature vectors; an LSTM-based speech encoder is built, inputting speech feature sequences into the LSTM network to obtain temporal feature representations of the speech; a BERT-based text encoder is built to encode text data and generate text feature vectors; and a GRU-based time series encoder is built to extract trend and periodic features from time series data.

[0066] The features extracted by each modality encoder are fused to construct a temporal fusion predictor. For example, an attention fusion mechanism is used, where the features of each modality are concatenated and input into an attention layer to learn the weight coefficients of each modality feature. Then, a weighted sum is performed to obtain the fused feature, which is then input into a fully connected layer to output the real-time conversion rate of the prediction.

[0067] The model is trained using a multimodal training dataset. A loss function (such as Mean Squared Error (MSE) is set to calculate the error between the predicted and actual conversion rates). The model parameters are adjusted using an optimization algorithm (such as the Adam optimizer) to gradually reduce the loss function until the model converges. During training, a validation set is used to validate the model and prevent overfitting. The model structure and hyperparameters are adjusted based on the validation results. Finally, the trained model is tested and evaluated using a test dataset to obtain performance metrics such as mean squared error and coefficient of determination R. 2 (etc.) to determine the final multimodal encoder and temporal fusion predictor model.

[0068] In one possible implementation, the multimodal encoder and temporal fusion predictor are trained and converged using the multimodal training dataset and test dataset. Step S400 further includes step S410, constructing a training batch based on causal-conversion aligned samples, using the original segmented fragments as anchor samples, and selecting high-conversion fragments corresponding to the anchor samples or from the same scene as positive samples. Specifically, for each anchor sample (original segmented fragment), its product ID and scene feature information are extracted. Fragments with the same product ID or scene features as the anchor samples and with high conversion rates are retrieved from the dataset as positive samples. For example, in PyTorch, the dataset is traversed to filter samples that meet the criteria. The anchor samples and positive samples are combined into a batch for model training.

[0069] Step S420: Based on the causal probability network, locally replace controllable elements to generate counterfactual negative samples. The controllable elements include at least one of discount strength, product salience, frequency of rare word triggers, and peak speech rate, while maintaining other modal features unchanged. Specifically, replaceable controllable elements are determined according to the causal probability network. For each positive sample, one or more controllable elements are randomly selected for replacement using predefined replacement rules or a random replacement strategy. For example, the discount strength is replaced from "80% off" to "90% off," or the peak speech rate is replaced from "fast" to "normal." A feature preservation method is used to ensure that the replaced sample remains consistent with the original sample in other modal features, thus generating counterfactual negative samples.

[0070] Step S430: During model training, the predicted differences between the anchor sample and positive and negative samples are compared. By minimizing the difference between the anchor sample and positive samples and maximizing the difference between positive and negative samples, the model gradually learns to distinguish between high-quality and low-quality operational scenarios. Specifically, a contrastive learning loss function, such as triplet loss, is used to measure the difference between the anchor sample and positive and negative samples. In each iteration, the distance (such as Euclidean distance or cosine distance) between the anchor sample and positive and negative samples is calculated. An optimization algorithm is used to update the model parameters, minimizing the distance between the anchor sample and positive samples and maximizing the distance between positive and negative samples.

[0071] Step S440 involves training and validation at three time scales: micro-time window, medium-time window, and large-time window. During training, model parameters are adjusted based on the test and validation results, and the output predictions are calibrated to ensure consistency between the predicted values ​​and the actual conversion rates, thus achieving convergence. Specifically, training and validation sets are constructed for each time scale (micro-time window, medium-time window, and large-time window). During training, model performance is periodically evaluated on the validation set, and model parameters are adjusted based on validation metrics (such as mean squared error or accuracy). For example, if the validation metrics do not improve over multiple epochs, an early stopping mechanism is triggered. The model's output predictions are calibrated. For example, a temperature calibration method is used, dividing the model's output probability values ​​by a temperature parameter to rescale the probability distribution to more closely approximate the true distribution.

[0072] Step S500: Deploy the multimodal encoder and the temporal fusion predictor together. The multimodal encoder is used to perform multimodal data segmentation and encoding on the live broadcast. The temporal fusion predictor uses the segmented and encoded multimodal data to predict the conversion rate in real time and outputs the real-time conversion rate prediction result of the live broadcast.

[0073] Specifically, the trained multimodal encoder and temporal fusion predictor models are saved in a deployment-friendly format, such as TensorFlow's SavedModel format or PyTorch's torchscript format. Then, a model deployment tool (such as TensorFlow Serving or TorchServe) is used to deploy the model to a server, and the model service port, input / output formats, and other parameters are configured.

[0074] During the live stream, multimodal data from the live feed is acquired in real time and preprocessed. For example, for video images, one frame is extracted from the video stream per second, scaled to the required input size (e.g., 224×224) using OpenCV, and then normalized. For audio data, audio input from the microphone is acquired in real time, and MFCC features (e.g., 40-dimensional) are extracted per second using Librosa. For bullet screen text, bullet screen messages are acquired in real time, segmented, and converted into word vectors (e.g., using a pre-trained Word2Vec model). For time series data, metrics such as the number of viewers and likes are recorded in real time per second and then normalized.

[0075] The preprocessed multimodal data is organized and concatenated according to the model input requirements. For example, image features, speech features, text features, and time series features are concatenated sequentially into a single feature vector, which is then sent to the deployed model service interface. Upon receiving the data, the model service uses a multimodal encoder to encode each modality, obtaining the feature vectors for each modality. These vectors are then fused and predicted using a time-series fusion predictor, outputting the real-time conversion rate prediction for the current live stream. Finally, the prediction results are returned to the front-end display interface or business system for relevant personnel to view and make decisions.

[0076] This application's embodiments employ techniques such as analyzing factors affecting live streaming conversion rates, collecting samples from multiple scenarios, building sample clusters, analyzing multimodal influencing factors based on the correspondence between sample clusters and influencing factors, clarifying the causal probability relationship between changes in each modality's data and the real-time conversion rate, constructing a causal probability network, performing segmentation and labeling based on the causal probability network, forming multimodal training and testing datasets, building a model framework for a multimodal encoder and a temporal fusion predictor, training the model using the aforementioned training and testing sets until convergence, obtaining a trained model, and deploying the multimodal encoder and temporal fusion predictor in conjunction. The encoder segments and encodes the multimodal data of the live streaming screen, and the predictor uses the encoded data to predict the real-time conversion rate and output the results. These techniques solve the technical problem of insufficient prediction accuracy in existing real-time conversion rate predictions for clothing live streaming, achieving the technical effect of accurate prediction of real-time conversion rates for clothing live streaming.

[0077] In the above text, refer to Figure 1 This paper describes in detail a multimodal deep learning method for predicting real-time conversion rates in live-streaming apparel sales according to embodiments of the present invention. Next, we will refer to... Figure 2 This invention describes a multimodal deep learning-based real-time conversion rate prediction system for live-streaming apparel sales, according to embodiments of the present invention.

[0078] The multimodal deep learning-based real-time conversion rate prediction system for live-streaming clothing according to embodiments of the present invention addresses the technical problem of insufficient prediction accuracy in existing live-streaming clothing conversion rate prediction systems, achieving the technical effect of accurate prediction of live-streaming clothing conversion rates. The multimodal deep learning-based real-time conversion rate prediction system for live-streaming clothing includes: a multi-scene sample acquisition module 10, a causal probability network construction module 20, a segmentation and labeling module 30, a training module 40, and an association and deployment module 50.

[0079] The system comprises the following modules: a multi-scenario sample acquisition module 10, used to analyze the influencing factors of live streaming conversion rate and collect multi-scenario samples based on these factors to construct a sample cluster; a causal probability network construction module 20, used to perform multimodal analysis of the influencing factors based on the correspondence between the sample cluster and the influencing factors, determine the causal probability relationship between changes in each modality data and the real-time conversion rate, and construct a causal probability network; a segmentation and labeling module 30, used to perform segmentation and labeling based on the causal probability network to construct a multimodal training dataset and a test dataset; a training module 40, used to build the model framework of a multimodal encoder and a temporal fusion predictor, and to train and converge the multimodal encoder and temporal fusion predictor using the multimodal training dataset and the test dataset to obtain the multimodal encoder and temporal fusion predictor; and an association and deployment module 50, used to associate and deploy the multimodal encoder and temporal fusion predictor, wherein the multimodal encoder is used to segment and encode multimodal data of the live streaming screen, and the temporal fusion predictor uses the segmented and encoded multimodal data to predict the conversion rate in real time and output the real-time conversion rate prediction result of the live streaming.

[0080] The detailed description of the specific configuration of the multi-scenario sample collection module 10 is as follows: As mentioned above, to analyze the influencing factors of live streaming conversion rate, the multi-scenario sample collection module 10 may further include: a data classification unit for classifying data according to the live streaming data source coverage type, including video stream, audio stream, user behavior, product context, and environmental factors; a multi-granularity decomposition unit for performing multi-granularity decomposition on each data source type to obtain the data source granularity level; a same-family live streaming data extraction unit for extracting same-family live streaming data, including different conversion rate gradients; an influencing factor alignment unit for aligning influencing factors according to the data source granularity level based on the same-family live streaming data, and fitting the conversion rate fluctuation influence relationship; and a filtering unit for filtering the conversion rate fluctuation influence relationship according to a preset fluctuation threshold to obtain the influencing factors of the live streaming conversion rate.

[0081] The multi-scenario sample collection module 10, which obtains the influencing factors of the live streaming conversion rate, may further include: a multi-level influencing factor dictionary construction unit, which constructs a multi-level influencing factor dictionary according to the scenario layer, modality layer, and fine-grained element layer. The fine-grained elements include product salience, posture type, and color / texture features in the video modality; volume, speech rate, and emotional fluctuations in the audio modality; discount terms and scarce cue words in the text modality; interaction rate and month-on-month increase in add-to-cart in the behavioral modality; and price range, discount intensity, and inventory status in the product and context modality. A multi-level projection unit is used to project the influencing factors of the live streaming conversion rate in a multi-level manner according to the hierarchical feature tags in the multi-level influencing factor dictionary, forming a multi-level dictionary structure of live streaming conversion rate influencing factors.

[0082] The multi-scenario sample collection module 10, which constructs a sample cluster, may further include: a stratified sampling collection unit for performing stratified sampling collection in multiple scenarios based on a multi-level influencing factor dictionary, including multi-scale samples with micro-time windows, medium-time windows, and large-time windows; and an oversampling unit for oversampling scarce composite events based on multi-scale samples, introducing supplementary samples of similar scenarios for new anchors or new products, and constructing a multi-scenario sample cluster.

[0083] The detailed description of the specific configuration of the causal probability network construction module 20 is explained as follows: As mentioned above, the causal probability network construction module 20 may further include: a multimodal causal probability network establishment unit, which is used to establish a multimodal causal probability network based on the correspondence between sample clusters and influencing factors, using a time-series causal structure learning method. The nodes include each fine-grained element, and the edges include time dependencies and intermodal causal relationships; and a local average treatment effect estimation unit, which is used to introduce instrumental variables and proximal causal modeling, and based on the multimodal causal probability network, estimate the local average treatment effect of each fine-grained element on the real-time conversion rate, forming a causal probability network that can perform do-operations, used to derive the causal probability relationship between multimodal data changes and the conversion rate.

[0084] The detailed description of the specific configuration of the segmentation annotation module 30 is explained as follows: As mentioned above, based on the causal probability network, segmentation annotation is performed to construct a multimodal training dataset and a test dataset. The segmentation annotation module 30 may further include: a weakly supervised segmentation annotation unit used to perform weakly supervised segmentation annotation on multimodal data based on video object detection, audio event detection, and text keyword matching, and to map the segmentation results to causal probability network nodes to construct causal relationships in the segmented data; and a sample dataset construction unit used to map and annotate the causal relationships in the segmented data with the sample conversion rate relationship to construct a sample dataset, divide the training samples and test samples according to a preset ratio, and establish sample time-series labels according to the temporal relationship of multi-scale samples to obtain the training dataset and the test dataset.

[0085] The process involves weakly supervised segmentation and annotation of multimodal data based on video object detection, audio event detection, and text keyword matching. The segmentation results are then mapped to causal probability network nodes to construct causal relationships within the segmented data. The weakly supervised segmentation and annotation unit may further include: a live stream segmentation subunit for segmenting the live stream based on the influence of video object detection and conversion rate, identifying salient product areas, human postures, and display actions; a live stream audio segmentation subunit for segmenting the live stream audio based on the influence of audio event detection and conversion rate, identifying speech rate, volume changes, and sound effects; a segmentation subunit for segmenting the live stream speech recognition results and bullet screen text based on the influence of text keyword matching and conversion rate, identifying discount information, scarce prompts, and purchase intent phrases; and a causal relationship labeling subunit for determining the influence relationship labels between segmented data features and conversion rate based on the mapping of the segmentation results to the causal probability network node states, thus identifying the causal relationships within the segmented data and obtaining the causal relationships of the segmented data.

[0086] Specifically, based on the causal relationship of the segmented data, a mapping and labeling process is performed with the sample conversion rate relationship to construct a sample dataset. Training and testing samples are divided according to a preset ratio, and temporal labels are established based on the temporal relationship of multi-scale samples to obtain the training and testing datasets. The sample dataset construction unit may further include: a causal-conversion aligned sample mapping subunit for mapping the causal relationship labels of segmented segments with the conversion rate labels of corresponding time windows to form causal-conversion aligned samples; a dataset partitioning subunit for partitioning the aligned samples into training and testing sets according to a preset ratio, and avoiding the simultaneous appearance of samples from the same broadcaster or the same scene in both the training and testing sets; a multi-scale temporal sample set establishment subunit for organizing the training and testing sets according to micro, medium, and large time windows to establish a multi-scale temporal sample set; and a temporal label generation subunit for generating temporal labels for continuous windows within each scale to support multi-scale temporal fusion modeling.

[0087] The detailed description of the specific configuration of the training module 40 is explained as follows: As mentioned above, the multimodal encoder and temporal fusion predictor are trained and converged using the multimodal training dataset and test dataset. The training module 40 may further include: a training batch construction unit for constructing training batches based on causal-conversion aligned samples, using the original segmented segments as anchor samples, and selecting high-conversion segments of the corresponding products or the same scenario as the anchor samples as positive samples; and a counterfactual negative sample generation unit for generating counterfactual negative samples by locally replacing controllable elements based on the causal probability network, wherein the controllable elements include discount intensity, product salience, and scarce prompt words. The model employs at least one of trigger frequency and peak speech rate to maintain other modal features unchanged. The prediction difference comparison unit compares the prediction differences between anchor samples and positive and negative samples during model training. By minimizing the difference between anchor samples and positive samples and widening the difference between positive and negative samples, the model gradually learns to distinguish between high-quality and low-quality operational scenarios. The training and validation unit trains and validates the model according to three time scales: micro-time window, medium-time window, and large-time window. During training, the model parameters are adjusted based on the test and validation results, and the output prediction results are calibrated to ensure consistency between the predicted values ​​and the actual conversion rates, thereby achieving convergence.

[0088] The multimodal deep learning-based real-time conversion rate prediction system for live clothing streaming provided in this invention can execute the multimodal deep learning-based real-time conversion rate prediction method for live clothing streaming provided in any embodiment of this invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0089] Although this application makes various references to certain modules in the system according to the embodiments of this application, any number of different modules can be used and run on user terminals and / or servers. The various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy distinction between each other and are not used to limit the scope of protection of this invention.

[0090] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application. In some cases, the actions or steps described in this application can be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A method for real-time conversion rate prediction for live fashion streaming using multi-modal deep learning, characterized in that, The application comprises the following steps: Analyzing the influencing factors of live conversion rate and collecting samples in multiple scenarios based on the influencing factors to construct a sample cluster; According to the correspondence between the sample cluster and the influencing factors, the influencing factors are analyzed in multiple modes, the causal probability relationship between the change of each mode data and the real-time conversion rate is determined, and a causal probability network is constructed; Based on the causal probability network, segmented labeling is performed to build a multi-modal training dataset and a test dataset; A model framework of multi-modal encoder and time series fusion predictor is built, and the multi-modal encoder and time series fusion predictor are trained and converged using the multi-modal training dataset and test dataset, thereby obtaining the multi-modal encoder and time series fusion predictor; The multi-modal encoder and time series fusion predictor are associated and deployed, wherein the multi-modal encoder is used for multi-modal data segmentation and encoding of live pictures, the time series fusion predictor uses the segmented and encoded multi-modal data to predict the real-time conversion rate, and outputs the live real-time conversion rate prediction result; Analyzing the influencing factors of live conversion rate, including: Classifying data according to live data source coverage types, including video stream, audio stream, user behavior, product context, and environmental factors; Multi-granularity decomposition is performed on each data source type to obtain data source granularity levels; Extracting live data of the same family, including different conversion rate gradients; Based on the live data of the same family, aligning the influencing factors according to the data source granularity levels, and fitting the conversion rate fluctuation influence relationship; According to the preset fluctuation threshold, the conversion rate fluctuation influence relationship is screened to obtain the influencing factors of the live conversion rate; Obtaining the influencing factors of the live conversion rate also includes: According to the scene layer, the mode layer, and the fine-grained element layer, a multi-level influencing factor dictionary is constructed, including product saliency, posture type, color / texture features in video mode, volume, speech rate, and emotional fluctuations in audio mode, preferential aperture, and scarcity prompt words in text mode, interaction rate, and incremental ratio in behavior mode, and price band, discount intensity, and inventory status in product and context mode; According to the hierarchical feature labels in the multi-level influencing factor dictionary, the influencing factors of the live conversion rate are projected in multiple levels to form a live conversion rate influencing factor with a multi-level dictionary structure.

2. The method of real-time conversion rate prediction for live clothing streaming using multi-modal deep learning according to claim 1, wherein, Constructing a sample cluster, including: Based on the multi-level influencing factor dictionary, hierarchical sampling is performed in multiple scenarios, including micro-time window, medium-time window, and large-time window multi-scale samples; According to the multi-scale samples, the rare compound events are oversampled, and the supplementary samples of similar scenarios of new anchors or new product introduction are constructed to form a multi-scenario sample cluster.

3. The method of claim 2, wherein, Constructing a causal probability network, including: According to the correspondence between the sample cluster and the influencing factors, a multi-modal causal probability network is established by using a time series causal structure learning method, the nodes include each fine-grained element, and the edges include time dependence relationship and inter-modal causal relationship; Introducing tool variables and proximal causal modeling, estimating the local average treatment effect of each fine-grained element on real-time conversion rate based on the multi-modal causal probability network, forming a do-operable causal probability network for deriving the causal probability relationship of multi-modal data changes on conversion rate.

4. The method of claim 2, wherein, Based on the causal probability network, segment annotation is performed to form a multi-modal training data set and a test data set, including: Weakly supervised segmentation annotation is performed on multi-modal data based on video object detection, audio event detection, and text keyword matching, and the segmentation results are mapped to the nodes of the causal probability network to construct the causal relationship of segmentation data; Based on the causal relationship of segmentation data, the relationship between sample conversion rate is mapped and labeled to construct a sample data set, and the training sample and test sample ratio is divided according to the preset ratio, and the sample time sequence label is established according to the time sequence relationship of multi-scale samples to obtain the training data set and test data set.

5. The method of real-time conversion rate prediction for live clothing streaming using multi-modal deep learning according to claim 4, wherein, Weakly supervised segmentation annotation is performed on multi-modal data based on video object detection, audio event detection, and text keyword matching, and the segmentation results are mapped to the nodes of the causal probability network to construct the causal relationship of segmentation data, including: Based on the influence relationship between video object detection and conversion rate, the live picture is segmented to identify the significant area of the commodity, the posture of the person, and the display action; Based on the influence relationship between audio event detection and conversion rate, the live audio is segmented to identify the speed, volume change, and sound effect event; Based on the influence relationship between text keyword matching and conversion rate, the live speech recognition result and the barrage text are segmented to identify the preferential range, the scarcity prompt word, and the purchase intention phrase; According to the mapping of the segmentation result on the node state of the causal probability network, the influence relationship label of the segmentation data feature and the conversion rate is determined, the causal relationship of the segmentation data is identified, and the causal relationship of the segmentation data is obtained.

6. The method of real-time conversion rate prediction for live clothing streaming using multi-modal deep learning according to claim 4, wherein, Based on the causal relationship of segmentation data, the relationship between sample conversion rate is mapped and labeled to construct a sample data set, and the training sample and test sample ratio is divided according to the preset ratio, and the sample time sequence label is established according to the time sequence relationship of multi-scale samples to obtain the training data set and test data set, including: According to the mapping of the causal relationship label of the segmentation segment and the conversion rate label of the corresponding time window, a causal-conversion alignment sample is formed; The training set and the test set are divided according to the preset ratio, and the same anchor or the same scene sample is avoided from appearing in the training set and the test set at the same time; The training set and the test set are respectively organized according to the micro time window, the medium time window, and the large time window to establish a multi-scale time sequence sample set; In each scale, a time sequence label is generated for a continuous window to support multi-scale time sequence fusion modeling.

7. The method of claim 6, wherein, The multi-modal encoder and the time sequence fusion predictor are trained and converged using the multi-modal training data set and the test data set, including: Based on the causal-conversion alignment sample, a training batch is constructed, the original segmentation segment is taken as an anchor sample, and a high-conversion segment corresponding to the anchor sample or the same scene is selected as a positive sample; According to the causal probabilistic network, counterfactual negative samples are generated by locally replacing controllable factors, including at least one of discount intensity, commodity salience, scarcity prompt word trigger frequency, and peak value of speech speed, to keep the remaining modal characteristics unchanged; During the model training process, the prediction differences between the anchor samples and the positive samples and the negative samples are compared, the differences between the anchor samples and the positive samples are minimized, and the differences between the positive samples and the negative samples are enlarged, so that the model gradually learns to distinguish between high-quality operation scenarios and degraded operation scenarios; According to the three types of time scales of micro time window, medium time window and large time window, training and verification are carried out, and in the training process, the model parameters are adjusted according to the test verification results, and the output prediction results are calibrated to ensure the consistency between the prediction value and the true conversion rate, and to achieve the purpose of convergence.

8. A multi-modal deep learning system for live conversion rate prediction for clothing live streaming, characterized in that, The system is used to implement the multi-modal deep learning real-time conversion rate prediction method of the clothes live broadcast according to any one of claims 1-7, and the system comprises: A multi-scene sample collection module is used to analyze the influencing factors of live broadcast conversion rate, and collect multi-scene samples based on the influencing factors to construct a sample cluster; A causal probabilistic network construction module is used to analyze the multi-modal of the influencing factors according to the corresponding relationship between the sample cluster and the influencing factors, determine the causal probability relationship between the change of each modal data and the real-time conversion rate, and construct a causal probabilistic network; A segmentation and labeling module is used to perform segmentation and labeling based on the causal probabilistic network, and to assemble a multi-modal training data set and a test data set; A training module is used to build a model framework of a multi-modal encoder and a time series fusion predictor, and to train and converge the multi-modal encoder and the time series fusion predictor using the multi-modal training data set and the test data set, to obtain the multi-modal encoder and the time series fusion predictor; An association deployment module is used to associate and deploy the multi-modal encoder and the time series fusion predictor, wherein the multi-modal encoder is used to perform multi-modal data segmentation and encoding on the live broadcast picture, and the time series fusion predictor is used to perform real-time conversion rate prediction using the segmented and encoded multi-modal data, and to output a live broadcast real-time conversion rate prediction result.

Images