Multimodal AI-based live broadcast advertisement effect real-time optimization method and system

Abstract

The invention provides a live broadcast advertisement effect real-time optimization method and system based on multi-modal AI, and relates to the technical field of advertisement optimization, and the method comprises the steps: collecting the multi-modal input information of a live broadcast scene, and extracting a feature vector after preprocessing; constructing a multi-modal-advertisement effect association graph, and integrating the multi-modal-advertisement effect association graph with the multi-modal knowledge graph to form a dynamically updated advertisement optimization knowledge network; based on real-time context awareness and a knowledge network, determining an optimal strategy through reinforcement learning evaluation sorting; the optimal strategy is converted into a specific instruction, and dynamic optimization is executed; collecting effect feedback data, and returning to a feature fusion and weight optimization link; the system comprises a multi-modal data acquisition and feature extraction module, a cross-modal feature alignment and knowledge network construction module, a reinforcement learning strategy generation and evaluation module and the like. According to the invention, the delay waiting time of the live broadcast advertisement optimization response is reduced, the real-time adaptability is improved, and the advertisement putting accuracy and conversion efficiency are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of advertisement optimization, in particular to a live broadcast advertisement effect real-time optimization method and system based on multi-modal AI. BACKGROUND

[0002] Under the background of the vigorous development of live e-commerce, live broadcast advertising has become one of the core channels of brand marketing. However, the live broadcast scene has high dynamics and uncertainty, and real-time changes in audience interest, environmental factors, etc. have a significant impact on advertising delivery effect. Existing live broadcast advertising optimization techniques mainly rely on historical data analysis and preset rules, which are difficult to cope with real-time challenges during live broadcast, resulting in large fluctuations in advertising effectiveness and loss of conversion opportunities. In the live broadcast product promotion scene, when the host recommends a product, the audience may express interest through multi-modal signals such as expressions, comments, and click behavior. However, existing technologies cannot capture and integrate these signals in real time to adjust advertising strategies in a timely manner. For example, in a beauty live broadcast, the audience shows positive emotions such as facial concentration and voice inquiries for a certain lipstick, but the click rate is low because the advertising content fails to highlight discount information or has poor display angles. Existing systems lack multi-modal conflict resolution capabilities and cannot identify such conflicts between intent and behavior, still relying on single-modal decision-making, missing conversion opportunities. In addition, existing optimization methods mostly use static knowledge graphs and offline update mechanisms, which cannot adapt to the real-time influx of live broadcast data. Knowledge graphs are constructed based on historical advertising data, with a long update cycle. When new hotspots or audience groups appear during live broadcast, strategy adjustments lag behind. At the same time, reinforcement learning models only use short-term indicators such as click rate as rewards, ignoring long-term user retention, which can easily lead to short-sighted strategies. These limitations make live broadcast advertising optimization fall into "local optimum" and make it difficult to balance real-time effectiveness and long-term goals. Therefore, there is an urgent need for a new live broadcast advertising effect real-time optimization method and system based on multi-modal AI. SUMMARY

[0003] The present application aims to provide a live broadcast advertising effect real-time optimization method and system based on multi-modal AI to solve the problems of advertising optimization response delay, strategy falling into local optimum, and difficulty in adapting to live dynamic scenarios caused by knowledge graph update lag, insufficient multi-modal feature fusion capability, and lack of real-time feedback loop in existing technologies. The specific technical solutions are as follows: The present application provides a live broadcast advertising effect real-time optimization method based on multi-modal AI, comprising: Step 1: Collect multi-modal input information of the live broadcast scene, and pre-process and extract features from the multi-modal input information to obtain multi-dimensional modal feature vectors. The multi-modal input information includes audience-side visual modal data, behavior modal data, and voice modal data, as well as advertising-side visual modal data, text modal data, and live room scene modal data. Step 2: Fuse the historical effect data of ad placement by cross-modal feature alignment and intent analysis, construct a multi-modal-ad effect correlation graph, and integrate the multi-modal-ad effect correlation graph with the multi-modal knowledge graph to form a dynamically updated ad optimization knowledge network; Step 3: Based on the live real-time context perception information and the ad optimization knowledge network, generate multiple groups of candidate optimization schemes, and use real-time click rate, real-time conversion rate, and real-time retention rate as reward signals to evaluate and rank the candidate optimization schemes through reinforcement learning algorithm to determine the optimal modal fusion and placement adjustment strategy; Step 4: Convert the optimal modal fusion and placement adjustment strategy into specific instructions for ad content switching, visual display optimization, placement timing adjustment, and interactive incentive configuration, and generate video frame replacement and subtitle update control parameters to drive the live system to perform dynamic optimization operations; Step 5: Collect real-time feedback data after adjustment and feed the feedback data back to the feature fusion and weight optimization stage, and realize closed-loop optimization combined with cross-modal conflict resolution rules.

[0004] Further, the multi-modal input information collected in step 1 includes: Real-time collection of audience-side visual modal data, behavior modal data, and voice modal data through multi-source sensors integrated in the live system, as well as ad-side visual modal data, text modal data, and live room scene modal data; The audience-side visual modal data is collected through the front camera of the user terminal to obtain the video stream of the audience's face, and is anonymized after obtaining user authorization; the behavior modal data is collected through the touch screen or interactive interface to obtain the operation sequence of the audience, including click frequency, scrolling speed, and dwell time; the voice modal data is collected through the microphone to obtain the voice audio stream of the audience, and is processed through voice-to-text or desensitization after authorization; The ad-side visual modal data is obtained through the ad server to obtain the image or video frame of the ad material; the ad-side text modal data is obtained by extracting the script content of the ad; The live room scene modal data is collected through environmental cameras or scene sensors, including background lighting, scene layout, and dynamic object information; All data are bound to a unified timestamp and are subjected to targeted preprocessing: visual data is subjected to illumination correction and inter-frame alignment, behavior data is subjected to smoothing filtering, voice data is subjected to noise reduction and frame processing, text data is subjected to word segmentation and removal of stop words, and scene data is subjected to spatial normalization.

[0005] Further, the feature extraction of the multi-modal input information in step 1 includes: For audience-side visual data and advertisement-side visual data, a CNN-ViT hybrid model is used for encoding to extract visual feature vectors; For speech data, a combination of MFCC and LSTM models is used for encoding to extract speech feature vectors; For advertisement-side text data, a BERT model is used for encoding to extract text feature vectors; For behavior data and scene data, a time-series feature encoding model is used to extract behavior feature vectors and scene feature vectors; Wherein, the CNN-ViT hybrid model uses ResNet-50 as the backbone network of the CNN part, the ViT part contains 6 layers of Transformer blocks, the number of heads of the multi-head self-attention layer is 8, and the attention hidden layer dimension is 768; The parameters of the MFCC feature extraction include a speech signal sampling rate of 16 kHz, a frame length of 25 ms, a frame shift of 10 ms, and an LSTM network containing 2 layers of hidden layers, each layer containing 256 hidden units; After all feature vectors are normalized to a unified dimension, they are combined into a multi-dimensional modal feature vector through concatenation operation.

[0006] Further, the step 2 of constructing the multi-modal-advertising effect association graph includes: The multi-dimensional modal feature vector is input into a cross-modal alignment module, an attention mechanism-based feature alignment network is used to calculate the correlation matrix between different modal features, and a multi-head attention layer is used to dynamically allocate the weights of each modal feature to highlight the key modal; The aligned feature vector is input into an intent analysis model to output user intent labels, including "high interest", "neutral" or "low interest"; Fusion of historical advertising effect data, including historical advertising click-through rate, conversion rate and retention rate indicators, to form an enhanced feature representation; Based on the enhanced feature representation, a multi-modal-advertising effect association graph is constructed, the nodes in the graph include advertising feature nodes, user preference nodes, effect indicator nodes and delivery parameter nodes, the edges represent the association relationship between the nodes, and the weights are generated by the attention mechanism.

[0007] Further, in step 2, the multi-modal-advertising effect association graph is integrated with the multi-modal knowledge graph, which includes: The pre-constructed entity types of the multi-modal knowledge graph include product category, user portrait, advertising feature, effect indicator, delivery parameter and scene feature; Subgraphs are extracted from the multi-modal-advertising effect association graph, and the similarity between the nodes of the subgraphs and the nodes of the multi-modal knowledge graph is calculated, with a similarity threshold of 0.7; If the similarity between the nodes in the association graph and the knowledge graph nodes is not less than 0.7, a mapping edge or a fusion node is established; After integration, the number of nodes of the knowledge network is updated to the number of multi-modal knowledge graph nodes plus the number of newly added nodes of the association graph, and the number of edges is updated to the number of multi-modal knowledge graph edges plus the number of association graph edges plus the number of mapping edges; When new data flows in, an incremental graph neural network is used to update the knowledge network, and the update process includes new entity / relationship addition, node weight adjustment, and invalid relationship deletion.

[0008] Further, the incremental graph neural network updates the knowledge network, including: New entity / relationship addition step: if an entity or relationship not contained in the multi-modal knowledge graph appears in the new data, use the BERT-NER model to extract the entity and add it as a new node, or use the REBEL model to extract the relationship and add it as a new edge; Node weight adjustment step: based on new effect data, use gradient descent method to fine-tune the embedding vector of the corresponding node, with a learning rate of 0.0005; Invalid relationship deletion step: if a relationship has a conversion rate of less than 1% for three consecutive times, the weight of the relationship is reduced to less than 0.1; if it is invalid for five consecutive times, the corresponding edge is deleted; The update frequency of the knowledge network is divided into two modes: basic update cycle every 10 minutes and instant update trigger.

[0009] Further, the generation of multiple groups of candidate optimization schemes in step 3 includes: Real-time receive current frame information of live data stream, including audience real-time interaction data, current advertising display state and live room real-time environment data, and generate real-time multi-dimensional context feature vector through feature extraction; Query the subgraph most relevant to the current context feature vector from the advertising optimization knowledge network, extract the implicit association rules and historical mapping relationships, and form the current optimization knowledge context; Input the real-time context feature vector and the optimization knowledge context into the scheme generator, which generates multiple groups of candidate optimization schemes containing content presentation parameters, delivery time period parameters and bidding strategy parameters based on the combination of rule and variational autoencoder generation model; Among them, the content presentation parameters include visual parameters, copy parameters and audio parameters; the delivery time period parameters are generated based on time point sequence; the bidding strategy parameters are generated based on real-time competitive environment data and knowledge network mapping relationship; After combining and excluding parameter conflicts, eight to ten groups of candidate optimization schemes are retained; The evaluation and ranking of candidate schemes through reinforcement learning algorithm includes: The reinforcement learning evaluation environment is constructed, the state space is the fusion vector of the real-time context feature vector and the optimization knowledge context, and the action space is the candidate optimization scheme vector; The reward function is designed as: reward value = ω1 x real-time click rate + ω2 x real-time conversion rate + ω3 x real-time retention rate, wherein ω1, ω2 and ω3 are weight coefficients, and are dynamically set according to the advertiser's target; The DQN model is used as the policy evaluation network, the DQN model includes three fully connected layers, the experience replay buffer size is 100000, the batch size is 64, the target network update frequency is once every 100 steps, the discount factor is 0.9, the learning rate is 0.001, and the ε-greedy strategy is used to select actions; The policy evaluation network outputs the expected reward estimate of each candidate scheme, and generates an ordered candidate scheme list in descending order of the estimate; Based on the ε-greedy strategy or directly selecting the scheme with the highest expected reward, the optimal modal fusion and delivery adjustment strategy is determined.

[0010] Further, the optimal strategy in step 4 is converted into specific instructions, which include: The optimal strategy parameters are decoded by the strategy instruction parser and mapped to content switching instructions, visual optimization instructions, timing adjustment instructions and incentive configuration instructions; The content switching instructions generate video frame replacement instructions and subtitle update instructions; the visual optimization instructions generate graphic rendering instructions; the timing adjustment instructions generate timing trigger instructions and bidding request instructions; and the incentive configuration instructions generate incentive trigger logic instructions; All underlying control parameters are packaged into control command frames and sent to the execution unit of the live streaming system through a low-latency communication link, including an advertisement player, a graphic rendering engine, a live streaming scheduler and an interactive component manager, to perform dynamic optimization operations; The real-time collection of the adjusted effect feedback data in step 5 includes: Collecting audience-side instant behavior data, advertisement-side performance data and live streaming room scene situation data, and binding timestamps and optimization strategy identifiers; The effect feedback data is input into the conflict detection module, and cross-modal conflict resolution is performed based on predefined rules; The predefined rules include: Rule one, when the visual modal data indicates that the user's interest is high but the behavior modal data shows that the click rate is decreasing, the behavior data is preferred, and the visual and behavior feature weights are adjusted; Rule two, when the short-term click rate reward and the long-term retention rate reward conflict, a discount factor is introduced to calculate the final reward value; Rule three, when the voice modal data and the text modal data conflict, the modal data with high confidence is preferred; Rule four, when the scene modal data indicates high conversion potential but the behavior modal data shows low conversion, prioritize the behavior data, adjust the scene feature weight; Rule five, when the advertisement text modal data conflicts with the visual modal data, prioritize the visual modal data, and generate text-visual alignment instructions; After conflict resolution, the purified feedback signal is output and fed back to the cross-modal feature alignment module, the advertisement optimization knowledge network of step 2, and the reinforcement learning strategy evaluation network of step 3 to realize closed-loop optimization.

[0011] The present application also provides a live broadcast advertisement effect real-time optimization system based on multi-modal AI, which is used to realize the method, and the system comprises: A multi-modal data acquisition and feature extraction module is used to acquire multi-modal input information of a live broadcast scene in real time through a multi-source sensor integrated in a live broadcast system, covering audience-side visual, behavior, and voice modal data, advertisement-side visual and text modal data, and live broadcast room scene modal data. The acquired data is preprocessed, visual features are encoded using a CNN-ViT hybrid model, text features are encoded using a BERT model, voice features are encoded using an MFCC and LSTM model, and behavior and scene data are processed using a time series feature encoding model to obtain multi-dimensional modal feature vectors. A cross-modal feature alignment and knowledge network construction module is used to fuse advertisement delivery historical effect data through cross-modal feature alignment and intent analysis, allocate feature weights by means of an attention mechanism, construct a multi-modal-advertisement effect correlation graph, combine a multi-modal knowledge graph to mine the implicit association between advertisement features and user preferences, the mapping relationship between delivery parameters and effect indicators, and form a dynamically updated advertisement optimization knowledge network. A reinforcement learning strategy generation and evaluation module is used to generate multiple groups of candidate optimization schemes containing content presentation, delivery time period, and bidding strategies based on live broadcast real-time situational awareness information and the optimization knowledge network, take real-time click rate, conversion rate, and retention rate as reward signals, evaluate and sort the candidate schemes through a reinforcement learning algorithm, and determine the optimal modal fusion and delivery adjustment strategy. An instruction analysis and control execution module is used to convert the optimal strategy into specific instructions for advertisement content switching, visual display optimization, delivery timing adjustment, and interactive incentive configuration, analyze and generate video frame replacement and subtitle update control parameters, and drive the live broadcast system to perform dynamic optimization operations. An effect feedback and closed-loop optimization module is used to acquire adjusted effect feedback data in real time, purify the data in combination with cross-modal conflict resolution rules, feed back the purified feedback signal to the feature fusion and knowledge network update link to realize millisecond-level closed-loop optimization.

[0012] Further, the multi-modal data acquisition and feature extraction module specifically comprises: A multi-source data collection unit is configured to collect visual, behavioral, and voice data of the audience end, visual and text data of the advertisement end, and live room scene data in real time through a front camera, a touch screen, a microphone, an advertisement server, and an environmental sensor of a user terminal, and to obtain user authorization and perform anonymization or desensitization processing before collection; A data preprocessing unit is configured to timestamp binding and targeted preprocessing of multi-modal data, including illumination correction and inter-frame alignment of visual data, smoothing filtering of behavioral data, noise reduction and frame division of voice data, word segmentation and removal of stop words of text data, and spatial normalization of scene data, to form a time-aligned multi-modal data stream. A multi-modal feature encoding unit is configured to extract visual features using a CNN-ViT hybrid model, voice features using a MFCC and LSTM combined model, text features using a BERT model, and behavioral and scene features using a time sequence encoding model, and to output multi-dimensional modal feature vectors normalized to a unified dimension.

[0013] An electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements the steps of the method when executing the program.

[0014] A computer-readable storage medium has a computer program stored thereon, and the program implements the steps of the method when executed by a processor.

[0015] The present application has the following beneficial effects: The present application constructs a complete real-time optimization system from multi-modal data perception, knowledge network construction, reinforcement learning decision-making to closed-loop feedback, and significantly improves the effect of live advertising. The core beneficial effects are as follows: First, by fusing visual, voice, text, behavior, and scene multi-modal data, and using cross-modal attention mechanism and dynamic knowledge network, a deeper and more comprehensive understanding of user intent and scene context is achieved, overcoming the limitations of single modal data; second, the real-time context is innovatively combined with the knowledge network rich in historical experience to drive reinforcement learning to generate optimization strategies, so that the decision-making responds to real-time changes and follows long-term rules, effectively avoiding short-sighted and local optimal strategies; finally, through a millisecond-level execution-feedback-conflict resolution closed loop, the system can continuously adapt and adjust to ensure the accuracy, real-time performance, and continuous optimization capability of the advertisement delivery, ultimately improving the click-through rate, conversion rate, and user retention rate.

[0016] The technical solutions of the present application will be further described in detail below with reference to the accompanying drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings: Figure 1 is a step schematic diagram of the live broadcast advertisement effect real-time optimization method based on multi-modal AI of the present application; Figure 2 is a structure schematic diagram of the live broadcast advertisement effect real-time optimization system based on multi-modal AI of the present application. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0019] In the embodiments of the present application, a live broadcast advertisement effect real-time optimization method based on multi-modal AI is provided, please refer to Figure 1 , the method comprises the following steps: Step 1: Collecting multi-modal input information of live broadcast scene, covering audience end visual, behavior, voice modal data, advertisement end visual, text modal data and live broadcast room scene modal data, after targeted preprocessing, using CNN-ViT hybrid model to encode visual features, BERT model to encode text features, MFCC and LSTM model to encode voice features, time sequence feature encoding model to process behavior and scene data, obtaining multi-dimensional modal feature vector; Specifically including the following steps: Step 101: Real-time collection of multi-modal input information of live scene by multi-source sensors integrated with live broadcast system. Audience-side data includes: video stream of audience's face collected by front camera of user terminal, which needs to be authorized by user before collection and uploaded after anonymization processing (such as face blur) in terminal or edge server, as visual modal data, used to capture audience's expression, attention direction, etc.; behavior modal data of audience collected through touch screen or interactive interface, such as click frequency, scrolling speed, dwell time, etc. Operation sequence; audio stream of audience's voice collected through microphone, which can be processed by speech-to-text in terminal after authorization, and only text features or desensitized audio features are uploaded, as speech modal data, used to analyze voice content and emotion; advertisement-side data includes image or video frame of advertisement material obtained through advertisement server, as advertisement visual modal data; extract the script content of the advertisement, as advertisement text modal data; live room scene data is collected by environment camera or scene sensor, including background light, scene layout and dynamic object information. All data are bound with a unified timestamp and subjected to targeted preprocessing: visual data is corrected by illumination and aligned between frames to eliminate jitter; behavior data is smoothed to remove outliers; voice data is processed by noise reduction and frame division to enhance clarity; text data is standardized by word segmentation and removal of stop words; scene data is normalized in space to ensure consistency. After preprocessing, time-aligned multi-modal data stream is formed as input for feature extraction. The real-time performance of the method is supported by a high-performance computing platform, which at least includes a GPU cluster or a dedicated AI processor (such as TPU) for accelerating inference calculation of large-scale models such as CNN, ViT, BERT, etc.; the live broadcast system accesses high-speed network link to ensure low-latency transmission of control instructions.

[0020] Step 102: Input the preprocessed multi-modal data into the corresponding feature extraction model. For audience-side visual data and advertisement-side visual data, a CNN-ViT hybrid model is used for encoding, and the specific parameters and training details of the model are as follows: The CNN part uses ResNet-50 as the backbone network, and the input image size of the network is 224x224x3 (RGB format). BatchNormalization technology is used in the network, and the batch size is set to 32; the activation function uses ReLU function; the pooling layer uses maximum pooling, and the pooling kernel size is 3x3 with a step of 2. The final output of the ResNet-50 backbone network is a local feature map with a size of 7x7x2048.

[0021] The input of the ViT part is the segmented and flattened feature map sequence. The ViT model includes 6 layers of Transformer blocks, each of which includes a multi-head self-attention layer and a Feed-Forward network. The number of heads of the multi-head self-attention layer is 8, and the attention hidden layer dimension is 768. The hidden layer dimension of the Feed-Forward network is 3072. The activation function uses the GELU function, and the dropout rate is set to 0.1.

[0022] The training process of the CNN-ViT hybrid model is divided into two stages: pre-training and fine-tuning. The pre-training stage is based on the ImageNet-1K dataset, and the fine-tuning stage is based on the live advertising visual dataset, which contains more than 50,000 advertising video frames and more than 100,000 audience face video frames. The training round of the fine-tuning stage is 50, the learning rate is 0.001 (the first 10 rounds use linear preheating, and the last 40 rounds use cosine decay), and the loss function uses the cross-entropy function (based on user interest label classification). The model finally outputs a 1024-dimensional visual feature vector through the global average pooling layer, which is normalized to the [0, 1] interval.

[0023] For speech data, the combination of MFCC and LSTM model is used for processing, and the specific parameters and training details are as follows: The parameter settings for MFCC feature extraction are as follows: the speech signal sampling rate is 16 kHz, the frame length is 25 ms, the frame shift is 10 ms, the pre-emphasis coefficient is 0.97, the number of Mel filter banks is 40, and the MFCC feature dimension is 13. The feature extraction process includes 1st order difference and 2nd order difference processing, and the final MFCC feature dimension is 39.

[0024] The structure parameters of the LSTM network are as follows: it contains 2 layers of hidden layers, each layer has 256 hidden units; the dropout rate is set to 0.2, and the recurrent dropout rate is set to 0.1; the input sequence length is 30 (corresponding to 3 seconds of speech); the output layer is a fully connected layer, and the activation function uses the ReLU function, finally outputting a 256-dimensional speech feature vector, which is normalized to the [0, 1] interval.

[0025] The training data of the LSTM model is the live audience speech dataset, which contains more than 20,000 audience speeches, each of which is labeled with "positive", "neutral", and "negative" emotion labels. The training process has 30 training rounds, the learning rate is 0.0005, the loss function uses the cross-entropy function, and the optimizer uses the Adam optimizer. When the validation set accuracy is not less than 85%, the training is stopped.

[0026] For the advertisement-side text data, the BERT model is used for encoding: the segmented text is input into the pre-trained BERT model, and the semantic features are extracted through multiple layers of Transformer blocks to generate a text feature vector. For behavior data and scene data, a time series feature encoding model (such as an LSTM or Transformer-based encoder) is used to extract the time series pattern, and the behavior data is encoded into a behavior feature vector to reflect the operation habits, and the scene data is encoded into a scene feature vector to depict the environmental dynamics; all feature vectors are normalized to a unified dimension.

[0027] Step 103: input the feature vectors of each modality extracted in step 102, including visual, text, speech, behavior, and scene feature vectors, into a feature fusion layer, which combines the feature vectors of different modalities into a unified multi-dimensional modal feature vector through a concatenation operation, with a dimension equal to the sum of the dimensions of each sub-feature. Before fusion, standardize each feature vector to ensure consistent dimensions, and generate a multi-dimensional modal feature vector as the basis data for subsequent cross-modal analysis, which is used to comprehensively represent the real-time state of the live streaming scene.

[0028] Step 2: through cross-modal feature alignment and intent analysis, fuse the historical effect data of advertisement delivery, allocate feature weights with the help of attention mechanism, construct a multi-modal-advertising effect association graph (as a real-time component of the advertising optimization knowledge network), combine the multi-modal knowledge graph to mine the implicit association between advertising features and user preferences, the mapping relationship between delivery parameters and effect indicators, and form a dynamically updated advertising optimization knowledge network; specifically including the following steps: Step 201: input the multi-dimensional modal feature vector into the cross-modal alignment module, which uses a feature alignment network based on attention mechanism: calculate the correlation matrix between different modal features, such as measuring the correlation strength between visual features and text features through cosine similarity; use a multi-head attention layer to dynamically allocate the weights of each modal feature, highlighting key modalities, such as when speech data dominates, increase the weight of speech features, the aligned feature vector is input into an intent analysis model, such as a neural network classifier, which outputs user intent labels through a fully connected layer and Softmax, including "high interest", "neutral" or "low interest", and analyzes the real-time inclination of the audience to the advertisement. At the same time, fuse the historical effect data of advertisement delivery, including the click-through rate, conversion rate, and retention rate indicators of historical advertisements, which are read from the database and time-aligned with the real-time feature vector to form an enhanced feature representation.

[0029] Step 202: Based on the aligned features and historical data, a multi-modal-advertising effect association graph is constructed. The nodes in the graph include advertising feature nodes, user preference nodes, effect indicator nodes, and delivery parameter nodes. The edges represent the association between nodes, and the weights are generated by the attention mechanism: the influence strength between nodes is calculated through the graph attention network, for example, the weight of the edge between the advertising feature and the user preference reflects the preference matching degree. The association graph is stored in a dynamic graph structure, the initial data comes from historical records, and the node representation is optimized through the Node2Vec graph embedding algorithm when updated in real time. In one specific embodiment, the advertising feature nodes include advertising material ID and copy keywords, the user preference nodes include intent labels and behavior patterns, the effect indicator nodes include click-through rate and conversion rate, and the delivery parameter nodes include bid and time period.

[0030] Step 203: Integrate the multi-modal-advertising effect association graph with the multi-modal knowledge graph to form a dynamically updated advertising optimization knowledge network; the construction and updating process of the multi-modal knowledge graph is as follows, and the entity types specifically include: the first type is product category, such as "lipstick category under makeup" and "mobile phone category under 3C"; the second type is user portrait, including "age layer (18-25 years old, 26-35 years old)" "consumption ability (high, medium, low)" "interest label (makeup, game, home appliance)"; the third type is advertising feature, including "visual style (high saturation, simple)" "copy type (promotion, answer, story)" "audio emotion (positive, neutral, negative)"; the fourth type is effect indicator, including "click-through rate (CTR)" "conversion rate (CVR)" "retention rate (RR)"; the fifth type is delivery parameter, including "bid amount" "delivery time period" "display duration"; the sixth type is scene feature, including "online number scale (less than 500, 500-2000, greater than 2000)" "light type (warm light, cold light, natural light)".

[0031] The first step of the pre-construction process is data collection, including live advertising historical data for the past twelve months, and a public advertising domain knowledge base. The live advertising historical data includes more than one million advertising delivery records, more than five million user behavior logs, and more than twenty thousand advertising material features. The public advertising domain knowledge base includes Ali Mama advertising knowledge graph and Baidu information flow advertising graph. The second step of the pre-construction process is entity extraction, using a combination of "rules + models": for structured entities such as product categories and delivery parameters, regular expressions are used for extraction, for example, extracting "10 yuan" as a delivery parameter entity from the expression "bid amount: 10 yuan"; for unstructured entities such as user portraits and advertising features, the BERT-NER model is used for extraction, which is pre-trained on advertising domain corpus, for example, extracting "22 years old" as an age layer entity and "lipstick" as an interest label entity from the comment "I am 22 years old and like lipstick". The third step of the pre-construction process is relation extraction, using a Transformer-based relation extraction model (such as the REBEL model), with the input content being "entity pair + context", for example, inputting the context "18-25 year old users click on high saturation visual ads", and extracting the relationship "18-25 year old user portrait entity prefers high saturation visual style advertising feature entity". The fourth step of the pre-construction process is graph initialization, using a Neo4j graph database to store entities and relationships, with the number of nodes in the initialized graph being no less than five hundred thousand, and the number of edges being no less than two million. The graph embedding model uses the TransE model, with an embedding dimension of 128, a training round of 100, and a learning rate of 0.001. The model maps entities and relationships into low-dimensional vectors, which are used for subsequent implicit association mining.

[0032] A subgraph is extracted from the multi-modal-advertising effect correlation graph constructed in step 202, which includes advertising feature nodes, user preference nodes, and effect indicator nodes, with one hundred to two hundred nodes of each type. The similarity between the nodes of the subgraph and the nodes of the multi-modal knowledge graph is calculated using the cosine similarity algorithm, with a similarity threshold of 0.7.

[0033] If the similarity between a user preference node (such as the "20-year-old female" node) in the correlation graph and a user portrait entity (such as the "18-25-year-old female" entity) in the knowledge graph is not less than 0.7, a "mapping edge" is established between the user preference node and the user portrait entity; if the similarity between an advertising feature node (such as the "red lipstick visual frame" node) in the correlation graph and an associated feature (such as the "lipstick category associated face-up effect frame") in the knowledge graph is not less than 0.7, the advertising feature node and the corresponding node of the associated feature are merged into the same node.

[0034] After integration, the number of nodes of the knowledge network is updated to "the number of nodes of the multi-modal knowledge graph + the number of newly added nodes of the association graph", wherein the newly added nodes of the association graph are nodes in the association graph with a similarity less than 0.7 to the nodes of the knowledge graph; and the number of edges of the knowledge network is updated to "the number of edges of the multi-modal knowledge graph + the number of edges of the association graph + the number of mapping edges", so as to ensure that the real-time data of the association graph seamlessly connects with the historical knowledge of the knowledge graph.

[0035] When new data (effect feedback data of step 5, newly collected multi-modal data) flows in, the knowledge network is updated using an incremental graph neural network (Incremental GNN), and the updating process includes three steps of new entity / relationship addition, node weight adjustment, and invalid relationship deletion. New entity / relationship addition step: if an entity (such as "new consumer brand lipstick") not contained in the multi-modal knowledge graph appears in the new data, the BERT-NER model is used to re-extract the entity and add it as a new node; if a relationship (such as "new consumer brand preference for Z generation users") not contained in the multi-modal knowledge graph appears in the new data, the REBEL model is used to extract the relationship and add it as a new edge.

[0036] Node weight adjustment step: based on new effect data (such as "the conversion rate of the advertising feature of high saturation visual style in the 26-35 age group of users is increased to 2.8%"), the embedding vector of the corresponding node is fine-tuned using the gradient descent method, the learning rate of this step is 0.0005, and the update step is once every 10 minutes. For example, based on the above new effect data, the relationship weight between "the user portrait entity of 26-35 years old prefers the advertising feature entity of high saturation visual style" is strengthened, and the weight value of this relationship is increased from the original 0.6 to 0.8.

[0037] Invalid relationship deletion step: if a relationship (such as "the scene feature of cold light type matches the advertising feature of the cosmetics category") shows a conversion rate less than 1% in three consecutive effect feedbacks, the weight of the relationship is reduced to below 0.1; if the relationship shows no effect in five consecutive effect feedbacks, the edge corresponding to the relationship is deleted to avoid invalid knowledge interfering with decision-making.

[0038] The updating frequency of the knowledge network includes two modes of basic updating period and real-time updating trigger: the basic updating period is once every 10 minutes; if there is sudden data (such as a 5% increase in the click rate of an advertisement), the updating is triggered to ensure the real-time nature of the knowledge network.

[0039] Meanwhile, a gradient boosting regression model (XGBoost model) is trained to learn the mapping relationship between the delivery parameters and the effect indicators. The input of the model is the "delivery parameter vector + knowledge network relationship weight", and the output is the predicted value of the effect indicator. The training data of the model is the "parameter-effect" paired data in the past month, and the sample size of the data is not less than one hundred thousand. The model iteratively trains new data every 24 hours, and the training process uses the relationship of the knowledge network (such as "the conversion rate corresponding to the bid amount greater than 10 yuan is not less than 2%") as a constraint condition to ensure that the model prediction logic is consistent with the logic of the knowledge network. The final knowledge network serves as the core knowledge base for subsequent scheme generation. In the prior art, the knowledge graph in the field of live broadcast advertisement optimization adopts a "static construction + offline update" mode, which only constructs a fixed graph based on historical data and cannot adapt to the real-time data change requirements in the live broadcast scene, resulting in knowledge lagging behind the actual scene. The knowledge network of the embodiment realizes the deep integration of historical knowledge and real-time data through a three-stage process of pre-construction, integration, and incremental GNN update: the offline pre-construction stage lays the knowledge foundation, the online integration stage connects real-time related data, and the real-time incremental update stage dynamically adjusts node weights and relationships, and can actively delete invalid relationships.

[0040] Step 3: Based on the live real-time situational awareness information and the optimization knowledge network, generate multiple candidate optimization schemes containing content presentation, delivery time period, and bidding strategy. Real-time click-through rate, conversion rate, and retention rate are used as reward signals to evaluate and rank the candidate schemes through reinforcement learning algorithm to determine the optimal modal fusion and delivery adjustment strategy; specifically including the following steps: Step 301: Real-time receive current frame information from live data stream, including audience real-time interaction data, current advertisement display state, and live room real-time environment data. Specifically, the audience real-time interaction data includes click stream sequence in the current time window, real-time comment sentiment polarity, and gift sending frequency; the current advertisement display state includes the ID of the currently playing advertisement material, the displayed duration, and the current interactive button click heat map; the live room real-time environment data includes real-time online number, anchor speaking speed, and background music rhythm. The above information is processed by the same feature extraction model as step 1 to generate real-time multi-dimensional situational feature vectors. At the same time, from the dynamically updated advertisement optimization knowledge network constructed in step 2, query the subgraph most relevant to the current situational feature vector, extract the implicit association rules and historical mapping relationships, and form the current optimization knowledge context.

[0041] Step 302: input the real-time context feature vector and the optimization knowledge into a scheme generator, which is a module combining rule-based and variational autoencoder (VAE) generation models. The rule base contains three types of core rule templates, and the generation model is based on a pre-trained VAE (training data is a live advertising material library for nearly 6 months, containing 100,000 valid samples). In a specific embodiment, the scheme generator first extracts the "context-strategy" mapping relationship from the optimization knowledge network. For example, "young user group + high online number" corresponds to the strategy direction of "prioritizing visual content + high-frequency bidding", and "late evening + low interaction" corresponds to the strategy direction of "prioritizing promotional copy + staggered delivery". Combined with the real-time context feature vector (such as the current user portrait label, live room online number, and interaction frequency), the scheme generator determines the specific optimization direction of the three adjustable strategy dimensions and excludes invalid dimensions. For example, if the current advertisement has played 80%, the "delivery time adjustment" dimension parameter is not generated. Optionally, the scheme generator calls the XGBoost model trained in step 2 to quickly predict the effect of the initially generated parameter combination, and filters out combinations with significantly lower predicted effects than the threshold, thereby improving the efficiency of subsequent VAE model filtering and reinforcement learning evaluation.

[0042] Based on the visual features of the current advertising material (such as the product category being beauty, the core visual elements being "product texture display" and "on-face effect"), the rule base outputs "visual element replacement rules". If the real-time context shows that "users are focusing on the face" (based on visual modality data), three sets of visual parameters are generated: the first set is to increase the product texture close-up frame ratio to 40%, with the original value being 20%; the second set is to add "host trial comparison" frames with an interval of one frame every 5 seconds; and the third set is to increase the brightness of high-saturation color product labels by 20%. Based on text modality data (such as comments containing "how much" and "how to buy"), the rule base outputs "copy style adjustment rules" and generates two sets of copy parameters: the first set is a promotional-oriented copy containing "limited time 30 minutes" and "click to get a 50 yuan discount"; and the second set is a question-answering-oriented copy containing "click to see the price" and "live room exclusive link". Based on voice modality data (such as user voice emotion being "frustrated"), the rule base outputs "audio rhythm rules" and generates two sets of audio parameters: the first set is to increase the tempo of the background music by 15%, with the original BPM being 80 and the adjusted BPM being 92; and the second set is to increase the host's voice speed by 10%, with the original value being 120 words per minute and the adjusted value being 132 words per minute. The above parameters are filtered through a pre-trained VAE model, which excludes parameter combinations with poor user historical feedback, and finally retains five sets of effective replacement content parameters.

[0043] The scheme generator first extracts the "time period-conversion" historical rules from the knowledge network, for example, the conversion rate corresponding to "30 seconds after the anchor explains the goods" is the highest, which is 3.5%; the conversion rate corresponding to "1 minute before the online number peak" is the second highest, which is 2.8%. Combined with the current live process (such as the current time point is "10 seconds after the product explanation"), three groups of time point sequences are generated: the first group of time point sequences is the current time point plus 20 seconds, the current time point plus 50 seconds; the second group of time point sequences is the current time point plus 40 seconds, the current time point plus 70 seconds; the third group of time point sequences is 60 seconds before the online number peak, 30 seconds before the online number peak, wherein the online number peak time is based on the online number trend prediction in the last 5 minutes. The time point sequence needs to meet the rule base constraint that "the advertisement interval is not less than 2 minutes", if there is a conflict in a group of time point sequences (such as the current time point plus 20 seconds and the interval of the last advertisement is less than 2 minutes), the group of time point sequences is excluded.

[0044] The scheme generator extracts key information from real-time competitive environment data (such as the average bid of the current bidding competitor is 10 yuan / 1000 displays), "bid-conversion" mapping relationship in the knowledge network (such as the conversion rate corresponding to the bid of 12 yuan is 2.5%, the conversion rate corresponding to the bid of 10 yuan is 1.8%), and combines the advertiser budget threshold (such as 15 yuan / 1000 displays) to generate three groups of bidding parameters: the first group is aggressive parameters, the bid is 13 yuan, and the bid increment per click is 0.5 yuan; the second group is stable parameters, the bid is 11 yuan, and the bid increment per click is 0.3 yuan; the third group is conservative parameters, the bid is 10 yuan, and the bid increment per click is 0.2 yuan.

[0045] In one specific embodiment, the scheme generator combines in the manner of "content presentation parameters (five groups) x delivery time period parameters (three groups) x bidding strategy parameters (three groups)", and excludes parameter conflict schemes (such as the combination of "aggressive bidding parameters + low conversion period parameters") in the combination process, finally retaining eight to ten groups of candidate optimization schemes. The vector format of each group of candidate optimization schemes is "[visual parameter ID, text parameter ID, audio parameter ID, time point sequence ID, bidding parameter ID]", which ensures that the parameter space covers different optimization directions such as "high input and high conversion", "low input and stable conversion", and "content optimization priority".

[0046] Step 303: build a reinforcement learning evaluation environment, the core of which is a policy evaluation network, which is based on DQN (DeepQ-Network) construction, and the specific parameters and training details are as follows: 1. State space and action space definition, state space is the fusion vector of real-time context feature vector and optimization knowledge context: real-time context feature vector contains 1024-dimensional visual features, 768-dimensional text features, 256-dimensional speech features, 512-dimensional behavior features, and 512-dimensional scene features, with a total dimension of 3072; the optimization knowledge context is the node embedding vector of the knowledge network, and the 20 most relevant nodes are selected, with each node having an embedding vector dimension of 128 and a total dimension of 2560. The total dimension of the fused vector after splicing is 5632, and the vector is normalized to the [0, 1] interval. The action space is the candidate optimization scheme vector generated in step 302, with eight to ten actions, each corresponding to a set of candidate optimization scheme parameters, and the action encoding uses a one-hot vector form.

[0047] 2. Reward function design, the calculation formula of the reward function is "reward value = ω1 × real-time click rate + ω2 × real-time conversion rate + ω3 × real-time retention rate", where the definitions and calculation methods of each parameter are as follows: real-time click rate is the ratio of the number of valid clicks within 3 seconds after the ad is displayed to the actual number of ad displays. The criteria for determining valid clicks exclude "false touch" clicks, and multiple clicks with a continuous click interval of less than 0.5 seconds are counted as only one valid click. The value range of real-time click rate is [0, 1]. Real-time conversion rate is the ratio of the number of target conversions within 10 seconds after the ad is displayed to the number of valid clicks. Target conversion is divided into three levels: first-level conversion is clicking on the ad and jumping to the product detail page, second-level conversion is adding the product to the shopping cart after jumping, and third-level conversion is completing payment after adding the product to the shopping cart. The specific selection of which level as the target conversion is determined according to the advertiser's goal. The value range of real-time conversion rate is [0, 1]. Real-time retention rate is the ratio of the number of valid users who remain online within 5 minutes after the ad ends to the number of online users at the time of ad display. The criteria for determining valid users exclude offline users, and users who do not interact within 5 minutes are not counted as valid users. The value range of real-time retention rate is [0, 1]. ω1, ω2, and ω3 are weight coefficients, and their sum is 1. The specific values are dynamically set according to the advertiser's goal: if the advertiser's goal is short-term traffic, ω1 takes 0.5, ω2 takes 0.3, and ω3 takes 0.2; if the advertiser's goal is medium-term conversion, ω1 takes 0.2, ω2 takes 0.6, and ω3 takes 0.2; if the advertiser's goal is long-term user retention, ω1 takes 0.1, ω2 takes 0.3, and ω3 takes 0.6. Real-time click rate, real-time conversion rate, and real-time retention rate are processed using Min-Max normalization, and the reference range for normalization is determined based on the effect data of similar ads in the past three months: the reference range for real-time click rate is [0.5%, 20%], the reference range for real-time conversion rate is [5%, 80%], and the reference range for real-time retention rate is [10%, 70%]. The normalization formula is "normalized value = (actual value - historical minimum value) / (historical maximum value - historical minimum value)".

[0048] 3. DQN model parameters and training process, the structure parameters of the DQN model are: containing 3 layers of fully connected layers, the first layer of fully connected layer has a hidden layer dimension of 2048, the second layer has 1024, and the third layer has 512; the activation function uses ReLU function; the output layer dimension is consistent with the action space size, and the activation function uses Linear function. The training parameters of the DQN model are: the experience replay buffer size is 100000, the batch size is 64; the target network update frequency is once every 100 steps; the discount factor γ is 0.9; the learning rate is 0.001, and the optimizer uses the RMSProp optimizer; the ε-greedy strategy is used to select actions, the initial value of ε is 0.9, which is attenuated by 0.01 every 1000 steps, and is kept stable after attenuation to 0.1. The training data of the DQN model is the "state-action-reward-next state" sequence, the collection frequency of which is one per second, and all sequences are stored in the experience replay buffer. The training of the DQN model adopts the combination of online incremental learning and regular retraining, the online incremental learning is based on the latest collected "state-action-reward" sequence, the network weight is fine-tuned once every 10 minutes, the learning rate is 0.0001, to quickly adapt to the changes of knowledge network and real-time feedback. Regular retraining is performed every morning, using all data in the past 24 hours, the iteration round is 50, and the training is stopped when the reward value of the validation set is improved by no less than 5%, to ensure the overall stability of the model.

[0049] The policy evaluation network receives a candidate solution vector as input, and fits the expected long-term reward of the solution executed in the current state through the above multi-layer fully connected network, first outputting the expected reward estimate of the solution. Subsequently, the system sorts all candidate solutions in descending order according to their expected reward estimates, generating an ordered candidate solution list.

[0050] Step 304: After obtaining the ordered list of candidate solutions, the policy selection module determines the final optimal modal fusion and delivery adjustment strategy according to the ε-greedy strategy or directly selects the solution with the highest expected reward. The optimal strategy contains a specific parameter set, indicating how to adjust the content, when to deliver, and what bidding strategy to use to compete in the current real-time situation. This optimal strategy will serve as the basis for subsequent generation of specific control instructions. The reinforcement learning solutions in the existing technology in the field of live broadcast advertising optimization are mostly trained based on real-time effect data, and only use click-through rate and conversion rate as reward signals, without considering the implicit association of multi-modal knowledge graph, resulting in the strategy generation being easily trapped in "local optimum", such as only pursuing short-term click-through rate improvement while ignoring long-term user retention rate. The reinforcement learning of the embodiment integrates "knowledge network context" into the state space, so that the strategy generation not only refers to real-time effect data, but also follows historical knowledge rules; at the same time, the reward function introduces purified data after multi-modal conflict resolution, ensuring data consistency and avoiding incorrect data guiding strategy deviation. In this way, the strategy generation can adapt to the real-time situation and also consider long-term optimization goals.

[0051] Step 4: Convert the optimal strategy into specific instructions for advertising content switching, visual display optimization, delivery timing adjustment, and interactive incentive configuration, parse and generate control parameters such as video frame replacement and subtitle update, and drive the live broadcast system to perform dynamic optimization operations; specifically including the following steps: Step 401: Receive the optimal modal fusion and delivery adjustment strategy, which is a structured parameter set. The strategy instruction parser in the system first decodes the strategy parameters and maps them to four executable operation instruction sets: content switching instructions, visual optimization instructions, timing adjustment instructions, and incentive configuration instructions. For content switching instructions, the parser determines the advertising material identifier that needs to be switched according to the content presentation dimension parameters in the strategy, and generates corresponding material loading instructions and scene transition instructions. For visual optimization instructions, the parser generates specific visual rendering parameters according to the visual display optimization parameters in the strategy, such as picture contrast enhancement, key information highlighting, layout adjustment, etc. For timing adjustment instructions, the parser calculates the precise advertising insertion time point based on the delivery timing parameters in the strategy and the timestamp of the live stream, and generates a timed trigger instruction. For interactive incentive configuration instructions, the parser generates configuration parameters for interactive components such as pop-up coupons, lottery activity start, and pop-up style change based on the intention to improve interaction in the strategy.

[0052] Step 402: Based on the four types of operation instructions, the control parameter generation module in the system further converts them into underlying control parameters recognizable by the live streaming system or the advertisement player. For content switching, a video frame replacement instruction is generated, which contains the address of the new advertisement video segment to be inserted in the content delivery network, the starting frame index, and the starting time code for replacing the original advertisement video. At the same time, a subtitle update instruction is generated, which contains the new advertisement text content, the subtitle display position, the font style, and the display duration. For visual display optimization, a graphics rendering instruction is generated, which changes the brightness, contrast, saturation of the advertisement display area or draws a highlight outline around specific product images by adjusting the shader parameters or calling the graphics processing interface. For timing adjustment, a precise timing trigger instruction is generated, which is sent to the live streaming controller to control the insertion or switching of the advertisement stream at the specified live streaming time point. At the same time, a bidding request instruction may be generated, which contains the bid amount determined in step 3 to participate in the competition for advertisement display opportunities in the real-time bidding system. For interactive incentive configuration, an incentive trigger logic instruction is generated, such as an instruction to automatically trigger the red envelope sending program when the audience sends a comment containing a specific keyword.

[0053] Step 403: All underlying control parameters are encapsulated into a unified control command frame and sent to the corresponding execution units of the live streaming system through a low-latency communication link. After receiving the video frame replacement and subtitle update instructions, the advertisement player completes the seamless switching of the advertisement content within the next video frame refresh period. The graphics rendering engine receives the graphics rendering instruction and adjusts the visual performance of the advertisement in real time. The live streaming scheduler receives the timing trigger instruction and controls the advertisement to be cut in and out at a time point accurate to milliseconds. The interactive component manager receives the incentive trigger logic instruction and configures and activates the corresponding interactive functions. All execution units return execution confirmation signals to the central controller after completion, ensuring that the optimization strategy is accurately translated into dynamic optimization operations of the system.

[0054] Step 5: Real-time collection of adjusted effect feedback data, backflow to feature fusion and weight optimization link, combined with cross-modal conflict resolution rules, to achieve millisecond-level closed-loop optimization, significantly improving the accuracy of advertisement placement and conversion efficiency; specific steps include the following: Step 501: After the dynamic optimization operation driven by step 4 is executed, the system immediately starts the effect monitoring process and collects multi-dimensional effect feedback data in real time. The collected data includes: Audience-side immediate behavior data, including click-through rate, conversion behavior, retention rate, emotional polarity change of interactive comments, and gift giving rate change within a predetermined time window after advertisement display.

[0055] Advertisement-side performance data, including actual complete play rate of the advertisement, time points of being skipped, and average viewing duration of users during the advertisement period.

[0056] Live room scene situation data includes the trend of online number change before and after advertisement playing, and the trend of sending frequency of barrage.

[0057] The above data is sampled at a high frequency, and is bound with corresponding time stamp and identification of executed optimization strategy, to form a context-based effect feedback data stream.

[0058] Step 502: Input the real-time collected effect feedback data stream into the conflict detection module. The module first compares the new feedback data with the existing knowledge in the advertisement optimization knowledge network constructed in step 2. When a conflict is detected, the conflict resolution module based on predefined rules is started. The output of this module will be used as an upper constraint to correct the data-driven attention weight distribution mechanism. In one specific embodiment, the predefined rules include: Rule 1, when visual modality data (such as facial expression recognition as "excited" "focused") indicates that the user's interest is high, but behavior modality data (such as click frequency <0.5 times / minute, stay time <30% of the total ad time) shows that the click rate is decreasing, the system gives priority to the behavior data as the final judgment basis, generates a weight adjustment signal: force the visual feature weight to be reduced to 40%-60% of the original weight, the behavior feature weight to be increased to 1.2-1.5 times of the original weight, covering the weight value calculated by the original attention network, the adjustment window is 5 subsequent advertisement display periods (about 30-60 seconds). Rule 2, when the short-term click rate reward (click rate within 3 seconds after ad display) and the long-term retention rate reward (user retention rate within 5 minutes after ad display) signals conflict (such as short-term click rate >15% but retention rate <5%), introduce a discount factor γ (γ value range 0.6-0.8, dynamically adjusted according to the advertiser's goal: short-term conversion goal takes 0.8, long-term user retention goal takes 0.6), the final reward value = γ × short-term click rate reward + (1-γ) × long-term retention rate reward, balance the two to ensure the long-term consistency of the optimization goal, the correction result of this rule needs to follow the direction of the advertiser's core optimization goal defined by the reward function weight. Rule 3, when the voice modality data (such as MFCC-LSTM model recognizing emotion as "positive", voice to text containing keywords such as "want" "not bad") and the text modality data (such as negative words in audience comments >40%, bullet screen containing expressions such as "not buy" "skip") conflict, the system calculates the confidence of the two modalities (voice confidence based on model output probability, text confidence based on comment sentiment polarity score), and preferentially adopts the modality data with higher confidence: if voice confidence >0.8 and text confidence <0.5, retain the voice modality conclusion; otherwise, retain the text modality conclusion; if both have a confidence between 0.5-0.8, take the weighted average of the two modality conclusions (weight according to confidence proportion). Rule 4, when scene modality data (such as live room online number >1000, background light is high saturation warm color) indicates "high conversion potential scene", but behavior modality data (such as conversion behavior occurrence rate <2%, interactive comment amount <10 / minute) shows low conversion, the system determines that the scene data has "noise interference" (such as online number contains a large number of hanging users), and preferentially adopts the behavior data, and temporarily adjusts the scene feature weight to 30%-50% of the original weight, while triggering scene data secondary filtering (remove hanging user data, correct light intensity detection error).Rule five, when the advertising text modal data (such as the copy contains "limited discount" "buy one get one free" and other promotional information) and the visual modal data (such as the advertising video frame does not show the discount label, the product details are blurred) conflict, the system gives priority to the "user can directly perceive" visual modal data (because visual information has a stronger impact on immediate decision-making), and generates "text-visual alignment instructions", which are fed back to the content switching module in step 4, requiring the addition of visual elements such as discount labels in the next advertising frame to achieve modal data consistency. During the resolution process, all rules are executed through the four steps of conflict type identification, confidence calculation, weight adjustment / conclusion selection and instruction feedback, and the purified feedback signal after consistency checking is output. Conflict resolution follows the following priority principles: data that directly reflects the user's final behavior (rules one, two and four) is given priority over data that reflects the user's subjective attitude (rules three and five); when rules have the same priority or applicability overlap, confidence calculation and weighted average are used as the final basis for decision.

[0059] Step 503: Purified feedback signals are fed back to the front-end feature fusion and weight optimization link of the system; Specifically, it includes feeding back to the cross-modal feature alignment and intent analysis module in step 2, using the data about the user's final behavior in the feedback signal to fine-tune the neural network parameters that allocate weights to each modality in the attention mechanism through online learning algorithms, so that feature fusion is more in line with actual results.

[0060] The feedback to step 2 builds an advertising optimization knowledge network, using this optimization strategy, real-time context and final effect as a new experience data to update the association weights between nodes in the graph through graph neural network incremental learning algorithm, for example, strengthening the "advertising feature-user preference" association edges that have been proven effective, and weakening the ineffective association edges.

[0061] The feedback to step 3 of the reinforcement learning strategy evaluation network stores the "state-action-reward" sequence of this decision in the experience replay buffer for subsequent regular retraining of the strategy evaluation network to optimize its accuracy in estimating the expected rewards of candidate solutions.

[0062] Steps 501 to 503 constitute a complete optimization closed loop, and the system continuously runs this loop with low delay under the support of the high-performance computing platform: performs optimization, collects feedback, resolves conflicts, and updates the model. Each closed loop makes the system's feature fusion more accurate, the knowledge network richer, and the strategy generation more effective, thereby realizing real-time, dynamic, and adaptive optimization of the advertising placement strategy in continuous live streaming, and ultimately significantly improving the accuracy and conversion efficiency of advertising placement. The existing live streaming advertising optimization solutions are mostly one-way processes, such as data collection, strategy generation, and execution, without a feedback closed loop, which cannot realize real-time adaptive adjustment, resulting in lagging optimization response and large fluctuation in effect. The embodiment realizes the closed loop process of execution, feedback, and update, ensures the consistency of feedback data by combining five types of cross-modal conflict resolution rules, has a small total delay of the closed loop, and updates the model covering feature weights, knowledge networks, and reinforcement learning networks, thereby ensuring that each closed loop can improve the optimization capability of the system.

[0063] The method embodiment realizes a complete real-time optimization closed loop by sequentially performing the following five core steps: multi-modal data collection and feature extraction, cross-modal feature alignment and knowledge network construction, reinforcement learning strategy generation and evaluation, instruction analysis and control execution, and effect feedback and closed loop optimization. The method has the beneficial effects that it deeply integrates multi-modal AI into the whole process of advertising optimization: in the initial stage, advanced models such as CNN-ViT and BERT are used to accurately extract multi-dimensional features, laying a data foundation for in-depth analysis; in the middle stage, an advertising optimization knowledge network that dynamically updates and integrates historical knowledge and real-time data is innovatively constructed, providing intelligent support for strategy generation; and in the final stage, through reinforcement learning evaluation and millisecond-level closed loop feedback, the advertising strategy can be adaptively adjusted based on real-time user reactions, thereby significantly improving the accuracy, real-time response capability, and ultimate conversion efficiency of advertising placement.

[0064] As shown in Figure 2 The embodiment of the present application also provides a live streaming advertising effect real-time optimization system based on multi-modal AI, which is used to realize the live streaming advertising effect real-time optimization method based on multi-modal AI. The system comprises: A multi-modal data collection and feature extraction module is used to collect multi-modal input information of a live streaming scene in real time through a multi-source sensor integrated in a live streaming system, covering audience-side visual, behavioral, and speech modal data, advertising-side visual, text modal data, and live streaming room scene modal data, and to perform targeted preprocessing on the collected data, encode visual features using a CNN-ViT hybrid model, encode text features using a BERT model, encode speech features using MFCC and LSTM models, and process behavioral and scene data using a time series feature encoding model, to obtain multi-dimensional modal feature vectors. The cross-modal feature alignment and knowledge network construction module is configured to fuse historical advertising effect data through cross-modal feature alignment and intent analysis, allocate feature weights by means of an attention mechanism, construct a multi-modal-advertising effect correlation graph, mine the implicit correlation between advertising features and user preferences and the mapping relationship between advertising parameters and effect indicators in combination with a multi-modal knowledge graph, and form a dynamically updated advertising optimization knowledge network. The reinforcement learning strategy generation and evaluation module is configured to generate multiple sets of candidate optimization schemes containing content presentation, advertising time period and bidding strategies based on live streaming real-time situational awareness information and the optimization knowledge network, evaluate and rank the candidate schemes by means of a reinforcement learning algorithm to determine an optimal modal fusion and advertising adjustment strategy, with real-time click rate, conversion rate and retention rate as reward signals. The instruction analysis and control execution module is configured to convert the optimal strategy into specific instructions for advertising content switching, visual display optimization, advertising timing adjustment and interactive incentive configuration, analyze and generate control parameters such as video frame replacement and subtitle update, and drive the live streaming system to perform dynamic optimization operations. The effect feedback and closed-loop optimization module is configured to collect real-time feedback data after adjustment, purify the data in combination with cross-modal conflict resolution rules, and return the purified feedback signals to the feature fusion and knowledge network update link to realize millisecond-level closed-loop optimization.

[0065] The multi-modal data acquisition and feature extraction module specifically includes: The multi-source data acquisition unit is configured to acquire audience-side visual, behavioral and voice data, advertising-side visual and text data, and live streaming room scene data in real time through a user terminal front camera, a touch screen, a microphone, an advertising server and an environmental sensor, and obtain user authorization and perform anonymization or desensitization processing before acquisition; The data preprocessing unit is configured to perform timestamp binding and targeted preprocessing on multi-modal data, including illumination correction and inter-frame alignment of visual data, smoothing filtering of behavioral data, noise reduction and frame division of voice data, word segmentation and stop word removal of text data, and spatial normalization of scene data, to form a time-aligned multi-modal data stream. The multi-modal feature encoding unit is configured to extract visual features by means of a CNN-ViT hybrid model, extract voice features by means of a MFCC and LSTM combined model, extract text features by means of a BERT model, and extract behavioral and scene features by means of a time series encoding model, and output multi-dimensional modal feature vectors normalized to a unified dimension.

[0066] The cross-modal feature alignment and knowledge network construction module specifically includes: The cross-modal alignment and intent analysis unit is configured to calculate a correlation matrix between different modal features, dynamically allocate feature weights by means of a multi-head attention mechanism, align multi-modal features, and output user intent labels based on a neural network classifier. A multi-modal-advertisement-effect correlation graph construction unit is configured to construct a dynamic correlation graph including advertisement feature nodes, user preference nodes, effect indicator nodes, and delivery parameter nodes based on the aligned features and historical effect data, and calculate correlation weights between nodes through a graph attention network; A knowledge network integration and update unit is configured to integrate the multi-modal-advertisement-effect correlation graph with a pre-constructed multi-modal knowledge graph, dynamically update node weights and relationships through an incremental graph neural network, and learn a mapping relationship between delivery parameters and effect indicators in combination with an XGBoost model.

[0067] The reinforcement learning strategy generation and evaluation module specifically includes: A real-time context perception unit is configured to receive current frame information of a live data stream, extract a real-time multi-dimensional context feature vector, and query a related subgraph and implicit association rules from a knowledge network; A candidate scheme generation unit is configured to generate multiple groups of candidate optimization schemes in three dimensions of content presentation, delivery time period, and bidding strategy based on a rule base and a variational autoencoder (VAE) generation model in combination with real-time context and knowledge context; A strategy evaluation and sorting unit is configured to construct a DQN-based strategy evaluation network, take real-time click rate, conversion rate, and retention rate as reward signals, estimate expected rewards of the candidate schemes, and sort them in descending order; A strategy selection unit is configured to determine an optimal modality fusion and delivery adjustment strategy from an ordered candidate scheme list according to an ε-greedy strategy or a highest reward principle.

[0068] The instruction analysis and control execution module specifically includes a strategy instruction analysis unit configured to decode optimal strategy parameters into four types of executable operation instructions of content switching, visual optimization, timing adjustment, and incentive configuration; A control parameter generation unit is configured to convert the operation instructions into underlying control parameters of video frame replacement, subtitle update, graphic rendering, timing triggering, bidding request, and interactive triggering; An instruction distribution and execution unit is configured to send control command frames to execution units such as an advertisement player, a graphic rendering engine, a live stream scheduler, and an interactive component manager through a low-latency communication link, and receive execution confirmation signals.

[0069] The effect feedback and closed-loop optimization module specifically includes: An effect data acquisition unit is configured to acquire audience-side behavior data, advertisement-side performance data, and live room scene situation data in real time, and form an effect feedback data stream with timestamps and strategy identifiers; A cross-modal conflict resolution unit is configured to detect and resolve conflicts between multi-modal data based on a pre-defined rule base, and output purified feedback signals; The model and knowledge backflow updating unit is configured to backflow the purified feedback signal to the feature alignment module, the knowledge network module and the reinforcement learning module, and optimize the feature weight, the knowledge network associated edge and the strategy evaluation network parameter through online learning and incremental updating algorithm.

[0070] Through the cooperative work of the above-mentioned modules, the system realizes the full-process automatic optimization from multi-modal data acquisition, real-time feature extraction, knowledge network construction, reinforcement learning strategy generation to instruction execution and feedback loop, and significantly improves the live advertising delivery accuracy and conversion efficiency.

[0071] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiments, which will not be described herein.

[0072] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application belong to the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. A method for real-time optimization of live advertising effects based on multi-modal AI, characterized in that, The method comprises the following steps: Step 1: Collecting multi-modal input information of live broadcast scenes, preprocessing and feature extraction of multi-modal input information, and obtaining multi-dimensional modal feature vectors; Wherein the multi-modal input information covers audience visual modal data, behavior modal data, speech modal data, advertisement visual modal data, text modal data, and live broadcast room scene modal data; Step 2: Through cross-modal feature alignment and intent analysis, fuse the historical effect data of advertisement delivery, construct a multi-modal-advertising effect correlation graph, integrate the multi-modal-advertising effect correlation graph with the multi-modal knowledge graph, and form a dynamically updated advertising optimization knowledge network; Step 3: Based on the real-time situational awareness information of live broadcast and the advertising optimization knowledge network, generate multiple candidate optimization schemes, and take real-time click rate, real-time conversion rate and real-time retention rate as reward signals, evaluate and sort the candidate optimization schemes through reinforcement learning algorithm, and determine the optimal modal fusion and delivery adjustment strategy; Step 4: Convert the optimal modal fusion and delivery adjustment strategy into specific instructions of advertisement content switching, visual display optimization, delivery time adjustment and interactive incentive configuration, and analyze and generate video frame replacement and subtitle update control parameters to drive the live broadcast system to perform dynamic optimization operation; Step 5: Real-time collection of effect feedback data, effect feedback data backflow to feature fusion and weight optimization link, combined with cross-modal conflict resolution rules, realize closed-loop optimization.

2. The method of claim 1, wherein, The multi-modal input information of live broadcast scenes in step 1 comprises: Real-time collection of audience visual modal data, behavior modal data, speech modal data, advertisement visual modal data, text modal data, and live broadcast room scene modal data through multi-source sensors integrated by the live broadcast system; Wherein the audience visual modal data is collected by the front camera of the user terminal to collect the video stream of the audience's face, and is anonymized after obtaining the user's authorization; the behavior modal data is collected by the touch screen or the interactive interface to collect the operation sequence of the audience, including click frequency, scrolling speed and dwell time; the speech modal data is collected by the microphone to collect the speech audio stream of the audience, and is processed by speech-to-text or desensitization after authorization; The advertisement visual modal data is obtained by the advertisement server to obtain the image or video frame of the advertisement material; the advertisement text modal data is obtained by extracting the script content of the advertisement; The live broadcast room scene modal data is collected by the environment camera or scene sensor, including background light, scene layout and dynamic object information; All data are bound with a unified timestamp and are subjected to targeted preprocessing: the visual data is subjected to illumination correction and interframe alignment, the behavior data is subjected to smoothing filtering, the speech data is subjected to noise reduction and frame processing, the text data is subjected to word segmentation and stop word removal, and the scene data is subjected to spatial normalization.

3. The method of claim 1, wherein, The feature extraction of multi-modal input information in step 1 comprises: For audience visual data and advertisement visual data, a CNN-ViT hybrid model is used for encoding to extract visual feature vectors; For speech data, a combination of MFCC and LSTM model is used for encoding to extract speech feature vectors; For the ad-side text data, a BERT model is used to encode and extract a text feature vector; For the behavior data and scene data, a time-series feature encoding model is used to extract a behavior feature vector and a scene feature vector; The CNN-ViT hybrid model uses ResNet-50 as the backbone network of the CNN part, and the ViT part contains 6 layers of Transformer blocks, with 8 heads in the multi-head self-attention layer and a hidden layer dimension of 768; the MFCC feature extraction parameters include a speech signal sampling rate of 16 kHz, a frame length of 25 ms, a frame shift of 10 ms, and an LSTM network containing 2 layers of hidden layers with 256 hidden units each; After all the feature vectors are normalized to a unified dimension, they are combined into a multi-dimensional modal feature vector through concatenation.

4. The method of claim 1, wherein, The construction of the multi-modal-ad effect association graph in step 2 includes: Inputting the multi-dimensional modal feature vector into a cross-modal alignment module, using an attention mechanism-based feature alignment network to calculate the correlation matrix between different modal features, and using a multi-head attention layer to dynamically allocate the weights of each modal feature to highlight the key modal; Inputting the aligned feature vector into an intent analysis model to output user intent labels, including "high interest", "neutral", or "low interest"; Fusing historical ad delivery effect data, including click-through rate, conversion rate, and retention rate indicators of historical ads, to form an enhanced feature representation; Based on the enhanced feature representation, a multi-modal-ad effect association graph is constructed, with nodes including ad feature nodes, user preference nodes, effect indicator nodes, and delivery parameter nodes, and edges representing the association between nodes, with weights generated by an attention mechanism.

5. The method of claim 4, wherein, In step 2, integrating the multi-modal-ad effect association graph with the multi-modal knowledge graph includes: The pre-constructed entity types of the multi-modal knowledge graph include product categories, user portraits, ad features, effect indicators, delivery parameters, and scene features; Extracting subgraphs from the multi-modal-ad effect association graph and calculating the similarity between the nodes of the subgraphs and the nodes of the multi-modal knowledge graph, with a similarity threshold of 0.7; If the similarity between a node in the association graph and a node in the knowledge graph is not less than 0.7, a mapping edge or a merged node is established; After integration, the number of nodes in the knowledge network is updated to the number of nodes in the multi-modal knowledge graph plus the number of newly added nodes in the association graph, and the number of edges is updated to the number of edges in the multi-modal knowledge graph plus the number of edges in the association graph plus the number of mapping edges; When new data flows in, an incremental graph neural network is used to update the knowledge network, and the update process includes new entity / relationship addition, node weight adjustment, and invalid relationship deletion.

6. The method of claim 5, wherein, The incremental graph neural network updates the knowledge network, including: New entity / relationship addition step: If an entity or relationship not included in the multi-modal knowledge graph appears in the new data, use the BERT-NER model to extract the entity and add it as a new node, or use the REBEL model to extract the relationship and add it as a new edge; Node weight adjustment step: Based on the new effect data, use the gradient descent method to fine-tune the embedding vector of the corresponding node with a learning rate of 0.0005; Invalid relationship deletion step: if a relationship shows a conversion rate less than 1% for three consecutive times, the weight of the relationship is reduced to less than 0.1; if it is invalid for five consecutive times, the corresponding edge is deleted; The update frequency of the knowledge network is divided into two modes: a basic update cycle every 10 minutes and an instant update trigger.

7. The method of claim 1, wherein, The generation of multiple groups of candidate optimization schemes in step 3 includes: Real-time reception of live data stream current frame information, including audience real-time interaction data, current advertisement display state and live room real-time environment data, and generation of real-time multi-dimensional context feature vector through feature extraction; Query the most relevant subgraph from the advertisement optimization knowledge network according to the current context feature vector, extract the implicit association rules and historical mapping relationship, and form the current optimization knowledge context; Input the real-time context feature vector and the optimization knowledge context into the scheme generator, which generates multiple groups of candidate optimization schemes containing content presentation parameters, delivery time period parameters and bidding strategy parameters based on the rule and variational autoencoder generation model combination; Among them, the content presentation parameters include visual parameters, text parameters and audio parameters; the delivery time period parameters are generated based on the time point sequence; the bidding strategy parameters are generated based on real-time competitive environment data and knowledge network mapping relationship; After combining and excluding parameter conflicts, eight to ten groups of candidate optimization schemes are retained; The evaluation and ranking of candidate schemes through reinforcement learning algorithm includes: Construct a reinforcement learning evaluation environment, with the state space being the fusion vector of the real-time context feature vector and the optimization knowledge context, and the action space being the candidate optimization scheme vector; The reward function is designed as: reward value = ω1 × real-time click rate + ω2 × real-time conversion rate + ω3 × real-time retention rate, where ω1, ω2, ω3 are weight coefficients, which are dynamically set according to the advertiser's target; DQN model is used as the policy evaluation network, which contains 3 layers of fully connected layers, the experience replay buffer size is 100000, the batch size is 64, the target network update frequency is every 100 steps, the discount factor is 0.9, the learning rate is 0.001, and the ε-greedy strategy is used to select actions; The policy evaluation network outputs the expected reward estimate of each candidate scheme, and generates an ordered candidate scheme list in descending order of the estimate; Based on the ε-greedy strategy or directly selecting the scheme with the highest expected reward, the optimal modality fusion and delivery adjustment strategy is determined.

8. The method of claim 1, wherein, The conversion of the optimal strategy into specific instructions in step 4 includes: Decode the optimal strategy parameters through the strategy instruction parser, and map them to content switching instructions, visual optimization instructions, timing adjustment instructions and incentive configuration instructions; Content switching instructions generate video frame replacement instructions and subtitle update instructions; visual optimization instructions generate graphics rendering instructions; timing adjustment instructions generate timing trigger instructions and bidding request instructions; incentive configuration instructions generate incentive trigger logic instructions; All bottom-level control parameters are packaged into control command frames, which are sent to the execution unit of the live system through a low-latency communication link, including the advertisement player, graphics rendering engine, live stream scheduler and interactive component manager, to perform dynamic optimization operations; The real-time collection of adjusted effect feedback data in step 5 includes: Collect audience end instant behavior data, advertisement end performance data and live room scene situation data, and bind timestamp and optimization strategy identification; Input the effect feedback data into the conflict detection module, and perform cross-modal conflict resolution based on the predefined rules; The predefined rules include: Rule one, when the visual modal data indicates that the user's interest is high but the behavior modal data shows that the click rate is decreasing, the behavior data is preferred, and the visual and behavior feature weights are adjusted; Rule two, when the short-term click rate reward and the long-term retention rate reward conflict, a discount factor is introduced to calculate the final reward value; Rule three, when the voice modal data and the text modal data conflict, the modal data with high confidence is preferred; Rule four, when the scene modal data indicates high conversion potential but the behavior modal data shows low conversion, the behavior data is preferred, and the scene feature weight is adjusted; Rule five, when the advertisement text modal data and the visual modal data conflict, the visual modal data is preferred, and a text-visual alignment instruction is generated; After conflict resolution, the purified feedback signal is output, and is fed back to the cross-modal feature alignment module, the advertisement optimization knowledge network of step 2 and the reinforcement learning strategy evaluation network of step 3, to realize closed-loop optimization.

9. A live advertising effect real-time optimization system based on multi-modal AI, for implementing the method of any one of claims 1-8, characterized in that, The system comprises: A multi-modal data acquisition and feature extraction module for collecting multi-modal input information of a live scene in real time through a multi-source sensor integrated in a live system, covering audience end visual, behavior, voice modal data, advertisement end visual, text modal data and live room scene modal data, and performing targeted preprocessing on the collected data, using a CNN-ViT hybrid model to encode visual features, a BERT model to encode text features, an MFCC and LSTM model to encode voice features, and a time series feature encoding model to process behavior and scene data, to obtain multi-dimensional modal feature vectors; A cross-modal feature alignment and knowledge network construction module for aligning cross-modal features and analyzing intentions, integrating advertisement delivery history effect data, assigning feature weights through an attention mechanism, constructing a multi-modal-advertisement effect correlation graph, combining a multi-modal knowledge graph to mine the implicit association between advertisement features and user preferences, the mapping relationship between delivery parameters and effect indicators, and forming a dynamically updated advertisement optimization knowledge network; A reinforcement learning strategy generation and evaluation module for generating multiple candidate optimization schemes containing content presentation, delivery time period and bidding strategy based on live real-time situational awareness information and optimization knowledge network, taking real-time click rate, conversion rate and retention rate as reward signals, evaluating and sorting the candidate schemes through a reinforcement learning algorithm, and determining the optimal modal fusion and delivery adjustment strategy; An instruction analysis and control execution module for converting the optimal strategy into specific instructions for advertisement content switching, visual display optimization, delivery timing adjustment and interactive incentive configuration, analyzing and generating video frame replacement and subtitle update control parameters, and driving the live system to perform dynamic optimization operations; An effect feedback and closed-loop optimization module for collecting real-time adjustment effect feedback data, purifying the data based on cross-modal conflict resolution rules, feeding back the purified feedback signal to the feature fusion and knowledge network update link, and realizing millisecond-level closed-loop optimization.

10. The system of claim 9, wherein, The multi-modal data acquisition and feature extraction module specifically comprises: A multi-source data acquisition unit, configured to acquire audience-side visual, behavioral, and voice data, advertisement-side visual and text data, and live room scene data in real time through a user terminal front camera, a touch screen, a microphone, an advertisement server, and an environmental sensor, and to obtain user authorization and perform anonymization or desensitization processing before acquisition; A data preprocessing unit, configured to perform timestamp binding and targeted preprocessing on multi-modal data, including illumination correction and inter-frame alignment of visual data, smoothing filtering of behavioral data, noise reduction and framing of voice data, word segmentation and removal of stop words of text data, and spatial normalization of scene data, to form a time-aligned multi-modal data stream; A multi-modal feature encoding unit, configured to extract visual features using a CNN-ViT hybrid model, extract voice features using a MFCC and LSTM combined model, extract text features using a BERT model, and extract behavioral and scene features using a time series encoding model, and output multi-dimensional modal feature vectors normalized to a unified dimension.

Images