Brand emotion marketing-oriented user consumption preference multi-modal representation learning and content generation method

Abstract

The application discloses a user consumption preference multi-modal representation learning and content generation method for brand emotional marketing, which realizes the joint modeling of user consumption preference and instantaneous emotion, interpretable strategy attribution and end-to-end automatic generation of personalized marketing content, and improves the precision and intelligent level of brand emotional marketing.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence and intelligent marketing technology, specifically to a method for learning and generating content based on multimodal representations of user consumption preferences for brand sentiment marketing. Background Technology

[0002] In the field of digital marketing, personalized advertising recommendation technology based on user emotional states has made some progress. Existing technologies collect multimodal data such as users' voice, text, facial expressions, and physiological signals to identify users' real-time emotional states and match or adjust advertising content accordingly. For example, the emotional recognition results can be used to dynamically replace visual elements, background music, or narration tone in advertising materials, or adjust the playback rhythm and narrative path of advertisements. In addition, some existing technologies have introduced federated learning frameworks and differential privacy protection mechanisms to collaboratively train marketing strategy models across multiple enterprise nodes, optimizing advertising delivery strategies while protecting user privacy.

[0003] However, the aforementioned existing technologies still have the following technical shortcomings in practical applications of brand emotion marketing.

[0004] First, existing technologies fail to structurally model the correlation between users' long-term consumption preferences and short-term emotional states. Most solutions merely retrieve matching content from a pre-made content library based on currently identified emotional tags, ignoring the impact of users' stable brand preferences and consumption habits on ad acceptance. Due to the lack of a quantitative representation of users' long-term preferences, the system cannot understand the structured needs of users with specific preferences in specific emotional states for marketing content, resulting in a mismatch between recommended ad content and users' deep-seated consumption psychology.

[0005] Second, the multimodal feature fusion process lacks interpretability. Existing technologies typically employ contrastive learning or attention mechanisms to align and weight features from different modalities, outputting a single sentiment label or matching score. This approach fails to provide brands with an interpretable attribution path, meaning it cannot distinguish whether a user's reaction is caused by momentary emotional fluctuations or long-term preference misalignments, leaving strategy optimization without a clear direction.

[0006] Third, the generation of personalized marketing content relies on the retrieval and splicing of pre-made material libraries. Current technologies for dynamically replacing materials and adjusting display parameters involve selecting from pre-made ad templates or sets of clips. This fails to generate entirely new, non-predefined visual content based on users' fine-grained preferences and real-time emotional states, thus limiting the creativity and adaptability of personalized marketing.

[0007] Fourth, the strategy optimization objective function is singular, focusing only on short-term interaction metrics. Existing reinforcement learning reward functions often target short-term behavioral indicators such as click-through rate, conversion rate, and dwell time, neglecting the balance between instantaneous user emotional satisfaction and long-term brand value. This optimization approach easily leads to strategies maximizing short-term gains at the expense of long-term user experience, hindering the establishment and maintenance of brand loyalty.

[0008] In summary, there is an urgent need in this field for a technology that can achieve joint modeling of users' long-term consumption preferences and short-term emotions, interpretable strategy attribution, and end-to-end automatic generation of personalized marketing content by constructing a heterogeneous hypergraph of sentiment and preferences, introducing a cascading diffusion model to achieve controllable content generation, and designing a multi-objective reinforcement learning optimization mechanism. Summary of the Invention

[0009] To overcome the problems of the prior art, this invention discloses a method for learning and generating content based on multimodal representations of user consumption preferences for brand emotional marketing.

[0010] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0011] This provides a method for learning and generating multimodal representations of user consumption preferences for brand sentiment marketing, including the following steps:

[0012] S1 collects the user's multimodal data in real time through a multimodal perception component deployed on the terminal device. The multimodal data includes voice signals, text data, facial expression video streams, and physiological signals.

[0013] S2, Construct a heterogeneous hypergraph. The node types of the heterogeneous hypergraph include: user preference nodes generated by mapping user historical interaction data in the semantic space of consumption preferences, real-time emotion nodes extracted by multimodal temporal convolutional networks based on multimodal data, and predefined brand attribute nodes.

[0014] The heterogeneous hypergraph is convolutionally processed by a hypergraph convolutional network to generate a joint conditional vector, which is formed by concatenating the updated user preference node embedding vector and the sentiment node embedding vector.

[0015] S3, input the joint conditional vector into the cascaded diffusion model, which includes a first stage and a second stage:

[0016] In the first stage, the layout diffusion model uses the joint conditional vector as a condition to generate a semantic layout map from pure noise through a multi-step denoising process. The semantic layout map uses different color blocks and contours to describe the macroscopic composition of the image to be generated.

[0017] In the second stage, the high-definition diffusion model uses semantic layout graph and joint conditional vector as guiding conditions to generate high-definition brand marketing images through multi-step denoising.

[0018] S4 employs a multi-objective reinforcement learning algorithm, using the weighted sum of user instantaneous emotional satisfaction and long-term brand value as the objective function to iteratively optimize the generation control parameters of the cascading diffusion model.

[0019] Preferably, the specific method for constructing the user preference node in step S2 is as follows:

[0020] Collect users' historical interaction data on the shopping platform, including purchase records, browsing history, favorites behavior, and text comments;

[0021] Each product or brand is mapped to a continuous consumer preference semantic space through a pre-trained variational autoencoder. The consumer preference semantic space includes design style dimension, functional value dimension, brand value dimension, and emotional association dimension.

[0022] The distribution center of the user's historical interaction items in the semantic space of consumption preferences is used as the long-term consumption preference embedding vector of the user, and this embedding vector is added to the heterogeneous hypergraph as the user preference node.

[0023] Preferably, the specific method for extracting real-time emotion nodes in step S2 is as follows:

[0024] Align the multimodal data temporally and feed it into a multimodal temporal convolutional network;

[0025] The multimodal temporal convolutional network outputs a triplet of the user's current emotional state, which includes pleasure, arousal and dominance. The triplet is then converted into an emotional node embedding vector and added to a heterogeneous hypergraph.

[0026] Preferably, the process of a hypergraph convolutional network performing convolution operations on heterogeneous hypergraphs includes:

[0027] For each hyperedge, the embedding vectors of all nodes connected by the hyperedge are aggregated using mean pooling or attention mechanism to generate the hyperedge feature vector.

[0028] For each node, collect the hyperedge feature vectors containing all hyperedges of that node, aggregate them again, and update the embedding representation of that node with a learnable weight matrix and bias parameters.

[0029] Preferably, the process of generating the semantic layout graph using the layout diffusion model in step S3 employs the following denoising iterative formula:

[0030]

[0031] Where, x tLet α be the noise latent code at step t, where t is the current time step. t For noise scheduling parameters, ᾱ t For cumulative product, ε θ For a noise predictor parameterized by U-Net or Diffusion Transformer, Ccon is the joint conditional vector, σ t Let ε be the standard deviation of the Gaussian noise, and ε be the random noise sampled from the standard normal distribution.

[0032] Preferably, in step S3, the high-definition diffusion model uses both the semantic layout graph and the joint condition vector as guiding conditions in the following specific way:

[0033] The semantic layout graph is downsampled to the same spatial resolution as the latent code of the high-definition diffusion model, and the downsampled semantic layout graph and the joint conditional vector are concatenated in the channel dimension as additional input to the noise predictor of the high-definition diffusion model.

[0034] Preferably, the objective function for multi-objective reinforcement learning in step S4 is:

[0035]

[0036] Among them, R emotion For instantaneous emotional rewards, the calculation is based on the real-time change in the user's emotional state after the generated high-definition brand marketing image is displayed.

[0037] R brand As a long-term brand value reward, it is obtained by weighting the user's repurchase probability and recommendation intention in a future preset period through a pre-trained user lifetime value prediction model.

[0038] R risk The strategy risk penalty is provided by the strategy risk score output by the hypergraph convolutional network, where α, β, and γ are preset weight coefficients.

[0039] Preferably, the strategy risk score is generated in the following way:

[0040] In the hypergraph convolutional network, for each candidate generation strategy corresponding to the hyperedge, the feature distance between the emotion node connected by the hyperedge and the brand attribute node is calculated. When the feature distance exceeds a preset threshold, a strategy risk score higher than the baseline value is output.

[0041] Preferably, the specific method for iteratively optimizing the generation control parameters of the cascade diffusion model in step S4 is as follows:

[0042] A policy network is trained using the PPO algorithm, with the policy network having a joint condition vector C. con As input, the output is the probability distribution of layout selection in the first stage of the cascade diffusion model and the local detail control parameters in the second stage;

[0043] Feedback data is obtained through user interaction. Based on the feedback data and the objective function, the total reward value R is calculated and the policy network parameters are updated to maximize the total reward value R under given constraints.

[0044] Preferably, it also includes a closed-loop feedback update step:

[0045] The new multimodal data generated after the high-definition brand marketing image generated in step S4 interacts with users is fed back to step S2 to update the user preference nodes and real-time sentiment nodes in the heterogeneous hypergraph, and steps S2 to S4 are re-executed to form a continuous iterative optimization loop.

[0046] The beneficial effects of this invention are as follows:

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] By constructing a heterogeneous hypergraph of emotions and preferences and performing hypergraph convolution, a structured joint modeling of users' long-term consumption preferences and short-term emotions is realized, overcoming the defect in existing technologies that preferences and emotions cannot be used in synergy, and enabling the generated content to simultaneously reflect users' stable consumption psychology and dynamic emotional changes.

[0049] By adopting a cascaded diffusion model and a two-stage architecture of semantic layout graph generation and high-definition image synthesis, controllable generation from joint conditional vectors to personalized marketing images is achieved, eliminating the reliance on pre-made material libraries and improving the flexibility of marketing content generation.

[0050] The design incorporates a multi-objective reinforcement learning objective function that includes instantaneous emotional rewards, long-term brand value rewards, and strategic risk penalties. This function can simultaneously optimize short-term interactive experience and long-term brand value, avoiding the problem of traditional methods that only pursue short-term metrics such as click-through rates and thus damage the long-term user experience.

[0051] By dynamically updating user preference nodes and real-time sentiment nodes in the hypergraph through a closed-loop feedback mechanism, the joint condition vector is continuously refined with each interaction, and the quality and personalization of the generated content are continuously improved in the iteration, forming a continuously optimized intelligent marketing closed loop. Attached Figure Description

[0052] Figure 1 This is a flowchart of a method for learning and generating content based on multimodal representations of user consumption preferences for brand sentiment marketing, as provided in Embodiment 1 of the present invention. Detailed Implementation

[0053] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, exemplary embodiments will be described in detail below, examples of which are illustrated in the accompanying drawings. In the following description relating to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of methods and systems consistent with some aspects of this application as detailed in the appended claims.

[0054] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a” and “the” as used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0055] The following detailed description of the specific implementation methods, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided in detail.

[0056] Example 1

[0057] Please refer to Figure 1 This embodiment provides a method for learning and generating multimodal representations of user consumption preferences for brand sentiment marketing, including the following steps:

[0058] S1, the method first collects the user's multimodal data in real time through a multimodal perception component deployed on the terminal device. The multimodal data includes voice signals, text data, facial expression video streams and physiological signals. The terminal device can be a smartphone, tablet or smart interactive screen. The multimodal perception component includes a microphone, camera and wearable physiological signal sensor. The collected data is then processed in the next step after being time-stamped and noise filtered.

[0059] Physiological signals were acquired using the following sensors:

[0060] The skin conductance sensor has a sampling frequency of 10Hz and its electrodes are attached to the inside of the user's wrist.

[0061] A photoplethysmography sensor with a sampling frequency of 64Hz is used to extract heart rate and heart rate variability.

[0062] A three-axis accelerometer with a sampling frequency of 32Hz detects subtle body movements of the user as auxiliary emotional signals.

[0063] The synchronization method for multimodal data is as follows:

[0064] All sensors are based on the system clock, each data point is accompanied by a hardware timestamp, the audio signal sampling rate is 16kHz, the facial expression video stream is 30fps, and physiological signals are collected at the above frequencies.

[0065] Data preprocessing includes framing the speech signal, with a frame length of 25ms and a frame shift of 10ms, and extracting 40-dimensional MFCC features;

[0066] Text data is processed by the BERT tokenizer;

[0067] 468 3D facial key points were extracted using MediaPipe to measure facial expressions.

[0068] Physiological signals are filtered by bandpass filtering to remove baseline drift.

[0069] S2, construct a heterogeneous hypergraph, which includes three types of nodes: user preference nodes generated by mapping user historical interaction data in the semantic space of consumption preferences, real-time emotion nodes extracted by multimodal temporal convolutional networks based on multimodal data, and predefined brand attribute nodes.

[0070] The process of constructing user preference nodes is as follows:

[0071] Collect users' historical interaction data on the shopping platform, including purchase records, browsing history, favorites behavior, and text comments;

[0072] Each product or brand is mapped to a continuous semantic space of consumer preferences through a pre-trained variational autoencoder.

[0073] This consumer preference semantic space includes dimensions of design style, functional value, brand value, and emotional association.

[0074] The distribution center of the user's historical interaction items in the semantic space is used as the long-term consumption preference embedding vector of the user, and this embedding vector is added to the heterogeneous hypergraph as the user preference node.

[0075] The process of extracting real-time sentiment nodes is as follows:

[0076] The multimodal data collected in step S1 is aligned by time and then input into a multimodal temporal convolutional network. The network outputs a triplet of the user's current emotional state, which includes pleasure, arousal and dominance.

[0077] The triplet is converted into an emotion node embedding vector and added to the heterogeneous hypergraph as a real-time emotion node.

[0078] Brand attribute nodes are predefined. Each brand is mapped to the same consumer preference semantic space as the user preference nodes according to its own characteristics, forming nodes with fixed coordinates.

[0079] After the heterogeneous hypergraph is constructed, a hypergraph convolutional network is used to perform convolution operations on the heterogeneous hypergraph. The convolution operation process includes:

[0080] For each hyperedge, the embedding vectors of all nodes it connects are aggregated using mean pooling or an attention mechanism to generate the hyperedge feature vector.

[0081] For each node, collect the hyperedge feature vectors containing all hyperedges of that node, aggregate them again, and update the embedding representation of that node through a learnable weight matrix and bias parameters.

[0082] The specific parameters of the hypergraph convolutional network are as follows:

[0083] The hyperedge construction rule is to construct a hyperedge based on all interactive events within a fixed time window, such as 30 minutes. This hyperedge connects the user preference nodes, real-time emotion nodes, and brand attribute nodes that are interacted with in the interaction. Hyperedges in adjacent time windows achieve time association by sharing emotion nodes.

[0084] The network consists of two stacked hypergraph convolutional layers, with each layer outputting an embedding dimension of 128.

[0085] The activation function is ELU after each convolutional layer.

[0086] The training loss function uses a contrastive loss to embed nodes within the same hyperedge closer together and nodes across different hyperedges farther apart. Specifically:

[0087]

[0088] Where L is the contrastive loss function value, a scalar, representing the overall penalty for the embedding distance between positive and negative sample pairs in the current batch of hyperedges;

[0089] E is the set of positive sample edges, that is, the set of node pairs corresponding to the actual hyperedges in the heterogeneous hypergraph. For each hyperedge, all the nodes connected by the hyperedge are paired up to form a node pair (u,v). All the node pairs generated by the hyperedges together form E.

[0090] ∉E is the set of negative sample edges, that is, the set of node pairs that do not exist in the heterogeneous hypergraph. During the training process, node pairs with the same number of positive sample edges are randomly sampled from all nodes as negative samples, and these node pairs are required to be not in E.

[0091] (u,v) represents a node pair, where u and v are two different nodes in a heterogeneous hypergraph, which can be any two of the following: user preference nodes, emotion nodes, or brand attribute nodes.

[0092] h uThis is the embedding vector of node u after forward propagation through the hypergraph convolutional network, with a dimension of 128. This vector is learned by the hypergraph convolutional network based on the node's initial features and the hypergraph structure.

[0093] h v Let h be the embedding vector of node v, with dimensions equal to h. u same;

[0094] ||h_u-h_v||² is the square of the Euclidean distance between the embedding vectors of node u and node v, which is the sum of the squares of the differences in each dimension of the two vectors. The smaller the value, the more similar the two nodes are, and the larger the value, the less similar they are.

[0095] The margin is a preset distance interval threshold with a value of 1.0. This threshold defines the expected embedding distance boundary between positive and negative sample pairs. The squared embedding distance of a positive sample pair should be less than the margin, and the squared embedding distance of a negative sample pair should be greater than the margin.

[0096] max(0,·) is the hinge loss function. It takes this value when the expression inside the parentheses is greater than 0, otherwise it takes 0. This function guarantees that loss is only generated when the deviation between the predicted distance and the expected distance is positive.

[0097] Optimizer, Adam, learning rate 0.0005, training for 100 epochs, batch size=64;

[0098] The parameters of the attention mechanism, during attention aggregation, are calculated using a single-layer fully connected network for the attention weights:

[0099]

[0100] Where W is a learnable matrix of 128×128, and v is a learnable vector of 128 dimensions;

[0101] After the convolution operation is completed, the updated user preference node embedding vector and the sentiment node embedding vector are concatenated along the feature dimension to generate a joint conditional vector.

[0102] S3, input the joint conditional vector into the cascaded diffusion model, which includes a first stage and a second stage;

[0103] In the first stage, the layout diffusion model uses the joint conditional vector as a condition to generate a semantic layout graph from pure noise through a multi-step denoising process. The semantic layout graph uses different color blocks and outlines to describe the macroscopic composition of the image to be generated. For example, blue color blocks represent the product display area, green color blocks represent the background atmosphere area, and line outlines represent the interaction positions between the product and the user.

[0104] The layout diffusion model is trained using a large dataset of advertising images labeled with layout categories, enabling it to learn to generate reasonable spatial composition based on joint conditional vectors.

[0105] In the second stage, the high-definition diffusion model uses the semantic layout graph generated in the first stage and the joint conditional vector as guiding conditions to generate a high-definition brand marketing image through multi-step denoising.

[0106] The semantic layout graph is downsampled to the same spatial resolution as the latent code of the high-definition diffusion model, and the downsampled semantic layout graph and the joint conditional vector are concatenated in the channel dimension as additional input to the noise predictor of the high-definition diffusion model.

[0107] The noise predictor uses a U-shaped network structure, which simultaneously references the global conditional vector and spatial layout constraints in each denoising iteration to gradually refine image details and finally outputs a high-definition brand marketing image with a resolution of no less than 1024×1024 pixels.

[0108] S4 employs a multi-objective reinforcement learning algorithm, using the weighted sum of user instantaneous emotional satisfaction and long-term brand value as the objective function, to iteratively optimize the generation control parameters of the cascade diffusion model;

[0109] The policy network is trained using a proximal policy optimization algorithm. The policy network takes the joint condition vector as input and outputs the probability distribution of layout selection in the first stage of the cascade diffusion model and the local detail control parameters in the second stage.

[0110] Reward signals are obtained through user interaction with the generated images. These reward signals consist of a weighted sum of instantaneous emotional rewards, long-term brand value rewards, and strategic risk penalties.

[0111] The strategy network updates its own parameters based on the reward signal to maximize the value of the weighted reward function under given constraints, thereby continuously optimizing the quality and adaptability of the subsequently generated high-definition brand marketing images.

[0112] Specifically, through the continuous execution of steps S1 to S4, this embodiment achieves a complete automated process from user multimodal data collection and joint representation learning of consumption preferences and emotions to the generation of personalized brand marketing content. This method organically integrates users' long-term consumption preferences with short-term emotional states and utilizes a cascading diffusion model to generate high-definition images with reasonable structure and rich details, significantly improving the personalization and intelligence level of content generation in brand emotion marketing.

[0113] Furthermore, in step S2 above, the specific method for constructing the user preference node is as follows:

[0114] Collect users' historical interaction data on shopping platforms. This historical interaction data includes purchase records, browsing history, collection behavior, and text reviews. This data reflects users' long-term contact and reactions to different products or brands and is the basis for constructing user preference nodes.

[0115] Each product or brand is mapped to a continuous semantic space of consumer preferences through a pre-trained variational autoencoder.

[0116] The structure of a variational autoencoder is as follows:

[0117] The encoder takes a text description vector of a product / brand as input, obtains 768-dimensional features through Sentence-BERT, passes it through three fully connected layers with 512, 256 and 128 nodes respectively, uses LeakyReLU activation function with a slope of 0.2, and outputs the mean μ and variance log(σ²), both with a dimension of 64.

[0118] The decoder samples from the latent variable z, which is 64-dimensional, from N(μ,σ²), and passes it through a three-layer fully connected network with 256, 512, and 768 nodes respectively. The activation function is ReLU, and the output is a reconstructed text feature vector.

[0119] The loss function is:

[0120]

[0121] Among them, L recon To reconstruct the mean square error between the input and the original input, L KL Let β be the KL divergence between the latent variable distribution and the standard normal distribution, with β = 0.001;

[0122] The training data consisted of text descriptions and multimodal content from 100,000 products / brands on an e-commerce platform. The training lasted 100 rounds with a batch size of 128, using the Adam optimizer and a learning rate of 0.001.

[0123] The output embedding, where the latent variable z represents the coordinates of the product / brand in the 64-dimensional semantic space of consumer preferences, can be interpreted through principal component analysis as four dimensions: design style, functional value, brand value, and emotional association, each with 16 dimensions.

[0124] The design style dimension measures the extent to which a product or brand ranges from minimalism to complexity; the functional value dimension measures the extent to which it ranges from traditional practicality to cutting-edge technology; the brand value dimension measures the extent to which it ranges from affordable and accessible to luxurious and high-end; and the emotional association dimension measures the extent to which it ranges from calm and restrained to vibrant and flamboyant.

[0125] The coordinates of each product or brand in this four-dimensional space are learned by a variational autoencoder based on a large number of labeled samples;

[0126] The distribution center of the user's historical interaction items in the semantic space of consumption preferences is used as the long-term consumption preference embedding vector of the user;

[0127] For all the user's historical interactions, extract the coordinates of the product or brand corresponding to each interaction in the semantic space of consumption preferences, calculate the average value of these coordinates in each dimension, and obtain a four-dimensional vector, which is the user's long-term consumption preference embedding vector.

[0128] This embedding vector quantitatively represents users' stable preference tendencies in four aspects: design style, functional value, brand value, and emotional association.

[0129] Finally, the embedded vector is added to the heterogeneous hypergraph as a user preference node. During the construction of the heterogeneous hypergraph, the user preference node, together with the real-time sentiment node and the brand attribute node, constitutes the node set of the hypergraph, which is used for subsequent hypergraph convolution operations and the generation of joint condition vectors. In this way, the user preference node is accurately embedded into the heterogeneous hypergraph, providing stable personalized constraints for subsequent content generation.

[0130] Furthermore, in step S2 above, the specific method for extracting real-time emotion nodes is as follows:

[0131] The multimodal data collected in step S1 are aligned by time, and the Mel frequency cepstral coefficient features of the speech signal are extracted in a frame of 25 milliseconds with a frame shift of 10 milliseconds to obtain a 40-dimensional feature vector for each frame.

[0132] Text data is converted into 768-dimensional feature vectors for each sentence through a pre-trained sentence embedding model and assigned to corresponding time points according to timestamps;

[0133] The facial expression video stream extracts the two-dimensional coordinates of 68 facial key points through a facial key point detection algorithm, and each frame yields a 136-dimensional feature vector.

[0134] Physiological signals include skin conductance response and heart rate variability. Numerical features are extracted at a sampling rate of once per second. Since the original sampling rates of each modality are different, a linear interpolation method is used to unify the feature sequences of all modalities to a time base of 30 frames per second, ensuring that the features of different modalities at the same time are strictly aligned in time.

[0135] The aligned multimodal data forms a multidimensional time series. Each time point contains speech feature vectors, text feature vectors, facial key point feature vectors, and physiological signal values. These feature vectors are concatenated to obtain the total input feature vector at that time point.

[0136] The aligned multimodal data is input into a multimodal temporal convolutional network, which consists of four stacked temporal convolutional blocks. Each temporal convolutional block contains a one-dimensional dilated convolutional layer, a batch normalization layer, and a modified linear unit activation layer.

[0137] The specific structure of the multimodal temporal convolutional network is as follows:

[0138] In the input layer, the features of each modality are aligned along the time axis. The total dimension of the feature vector at each time step is: speech 40 dimensions + text 768 dimensions + face 136 dimensions + physiology 2 dimensions = 946 dimensions.

[0139] The independent modality convolution branch first passes through an independent one-dimensional convolutional layer with a kernel size of 3, a stride of 1, and 32 output channels to extract modality-specific features. Then, all modality features are concatenated in the channel dimension to obtain a 160-dimensional feature sequence.

[0140] The four temporal convolutional blocks have output channels of 64, 128, 256, and 512 respectively. Each block contains dilated convolution with a kernel size of 3, batch normalization, Dropout (0.2), and ReLU activation, with dilation rates of 1, 2, 4, and 8 respectively.

[0141] Global average pooling is applied to the output of the last convolutional block, resulting in a 512-dimensional feature vector.

[0142] The output layer consists of three independent fully connected nodes, which output pleasure, arousal, and dominance respectively. The activation function is Tanh, and the output range is [-1, 1].

[0143] The training dataset used the CMU-MOSI and RECOLA multimodal sentiment datasets, with approximately 20,000 labeled segments. The labels were continuous values ​​for pleasure, arousal, and dominance. The Adam optimizer was used with a learning rate of 0.001, and the training lasted for 50 epochs. The loss function was the mean absolute error.

[0144] Post-processing involves smoothing the output values ​​using an exponentially weighted moving average with a smoothing coefficient of 0.3 to suppress noise.

[0145] The first convolutional block has a dilation rate of 1, the second convolutional block has a dilation rate of 2, the third convolutional block has a dilation rate of 4, and the fourth convolutional block has a dilation rate of 8. The kernel size of each convolutional layer is 3, and the number of output channels is 64, 128, 256, and 512 respectively.

[0146] During the training phase, the network uses a multimodal sentiment dataset labeled with pleasure, arousal, and dominance for supervised learning, and the loss function is the mean squared error loss.

[0147] After training, the network is able to extract deep temporal features from the input multidimensional time series and map them to the emotional state space.

[0148] The multimodal temporal convolutional network outputs a triplet of the user's current emotional state, which includes pleasure, arousal, and dominance.

[0149] Pleasure level represents the degree to which a user's emotions range from negative to positive, and its value ranges from negative one to positive one.

[0150] Arousal level represents the degree to which a user's emotions range from calm to excited, and its value ranges from negative one to positive one.

[0151] Dominance represents the strength of a user's sense of control over the current situation, and its value ranges from negative one to positive one.

[0152] These three values ​​are output by the last fully connected layer of the network and correspond to the three regression tasks, respectively;

[0153] The triples are converted into emotion node embedding vectors, and these vectors are added to the heterogeneous hypergraph. The conversion method involves arranging the three values ​​of pleasure, arousal, and dominance in sequence to form a three-dimensional vector. Then, a trainable single-layer linear mapping network is used to extend this three-dimensional vector to the same dimension as the user preference node embedding vector, such as 128 dimensions, to obtain the final embedding vector. This embedding vector serves as the real-time emotion node and together with the user preference node and brand attribute node, it constitutes the node set of the heterogeneous hypergraph.

[0154] In this way, the user's real-time emotional state is quantified into node embedding vectors that can participate in graph operations, providing dynamic emotional input for the subsequent generation of hypergraph convolution and joint conditional vectors.

[0155] Furthermore, during the process of performing convolution operations on heterogeneous hypergraphs using the aforementioned hypergraph convolutional network, the following specific operations are performed:

[0156] Traverse each hyperedge in the heterogeneous hypergraph. Each hyperedge connects multiple nodes, which include one or more of the following: user preference nodes, real-time sentiment nodes, and brand attribute nodes.

[0157] Extract the embedding vectors of all nodes connected by the hyperedge. These embedding vectors have the same dimension in the feature space. Aggregate these embedding vectors using mean pooling.

[0158] Calculate the arithmetic mean of the embedding vectors of all connected nodes across all dimensions to obtain a vector with the same dimensions as the embedding vector of a single node; this vector is the hyperedge feature vector. Alternatively, an attention mechanism can be used for aggregation: calculate a learnable attention weight for each connected node, the weight value is obtained by passing the node's embedding vector through a small fully connected network, and then sum the embedding vectors of all nodes according to the attention weights to obtain the hyperedge feature vector.

[0159] Both aggregation methods can obtain the hyperedge feature vector that represents the high-order correlation information contained in the hyperedge;

[0160] For each node in the heterogeneous hypergraph, traverse it. For the node being processed, collect all hyperedges in the heterogeneous hypergraph that contain that node, and extract the hyperedge feature vectors generated in the previous step.

[0161] These hyperedge feature vectors are then aggregated again, and the aggregation method can also be average pooling or attention mechanism.

[0162] During mean pooling, the arithmetic mean of all collected hyperedge feature vectors in each dimension is calculated to obtain the aggregate vector;

[0163] During the aggregation of attention mechanisms, attention weights are calculated for each hyperedge feature vector, and then the aggregated vector is obtained by weighted summation.

[0164] After obtaining the aggregated vector, a linear transformation is performed on the aggregated vector using a learnable weight matrix and learnable bias parameters. The transformation result is used as the updated embedding representation of the node.

[0165] The mathematical expression for this update process is that the new embedding vector equals the learnable weight matrix multiplied by the aggregation vector plus the learnable bias parameters;

[0166] The dimension of the learnable weight matrix matches the dimension of the node embedding vector, and the initial values ​​are randomly initialized and optimized through backpropagation during model training.

[0167] Specifically, by sequentially performing the above aggregation and update operations on all hyperedges and all nodes in the heterogeneous hypergraph, a round of hypergraph convolution is completed. Through the stacking of multiple layers of hypergraph convolution, the node embedding representation gradually integrates structural information from higher-order neighborhoods, enabling the updated user preference node embedding vector and emotion node embedding vector to fully reflect the complex interaction between user consumption preferences and real-time emotions. Finally, the updated user preference node embedding vector and emotion node embedding vector are concatenated to obtain the joint condition vector used for the subsequent cascaded diffusion model.

[0168] Furthermore, in step S3 above, the process of generating the semantic layout graph using the layout diffusion model employs the following denoising iterative formula for progressive denoising:

[0169]

[0170] The meanings, values, and calculation processes of each parameter in the formula are as follows.

[0171] Where, α t The noise scheduling parameters are set as follows: the total number of denoising steps T is set to 1000 steps, and the value of t is T, T-1, ..., 1. The noise scheduling parameter α is... t Linear scheduling method is used:

[0172]

[0173] Make α1 = 0.9999, α T =0.98, cumulative product ᾱ t Defined as all α from step 1 to step t i The product of these products is pre-calculated and stored before each iteration, i.e.:

[0174]

[0175] x t This represents the noise latent code at step t, with a spatial size of 64×64 pixels and 4 channels. The initial latent code is x. T Generated by random sampling from a standard normal distribution with a mean of 0 and a variance of 1;

[0176] C con The joint conditional vector has a dimension of 256 and is formed by concatenating the updated 128-dimensional user preference node embedding vector and the 128-dimensional sentiment node embedding vector.

[0177] ε θ A noise predictor parameterized by a U-Net, which includes an encoder, an intermediate block, and a decoder;

[0178] The encoder consists of four downsampling blocks, each containing two residual convolutional layers and a convolutional downsampling layer with a stride of 2, with channel numbers of 128, 256, 256, and 512 respectively.

[0179] The intermediate block contains two residual convolutional layers and one cross-attention layer. The cross-attention layer uses the intermediate feature maps of U-Net as queries and the joint conditional vector C. con As keys and values, they enable conditional guidance;

[0180] The decoder consists of four upsampling blocks, each containing two residual convolutional layers and a bilinear upsampling layer, with channel numbers of 512, 256, 128, and 64 respectively. The decoder output is passed through convolutional layers to obtain the signal with respect to x. t Noise prediction maps of the same size;

[0181] Time step t is sinusoidally position-encoded and then added to the bias term of the residual convolutional layer in U-Net to achieve temporal conditional injection;

[0182] The noise predictor is optimized by minimizing the mean squared error loss function during the training phase using a pair of layout graphs and a joint conditional vector dataset. The layout graphs in the training data are from a publicly available ad layout dataset, and the joint conditional vectors are extracted by a pre-trained hypergraph convolutional network.

[0183] σ t Let $\frac{ ...

[0184]

[0185] ε is random noise sampled from a standard normal distribution, and its size is related to x. t They are identical, and each element is independently and identically distributed;

[0186] In each denoising iteration, the current latent code x is first... t The current time step t and the joint condition vector C con Input noise predictor ε θ The predicted noise value is obtained.

[0187] Calculate x t Subtracting the scaled prediction noise, the scaling factor is: ;

[0188] Then divide the above result by The denoised principal components are obtained, and finally, the components are added by σ. t The random noise term consisting of ε and ε yields the latent code x for the next step. t-1 ;

[0189] Repeat the above iterative process until t=1 to obtain the final latent code x0;

[0190] x0 is input into a pre-trained decoder, which consists of three transposed convolutional layers that progressively upsample the latent code from 64×64×4 to 256×256×3 and output a color image with pixel values ​​between 0 and 1 via a sigmoid function. This image is the semantic layout map.

[0191] The semantic layout diagram uses different color blocks and outlines to describe the macroscopic composition of the image to be generated. For example, the background area, the main product area, and the text area are marked with different colors.

[0192] Specifically, through the above-mentioned denoising iterative formula and the corresponding network structure and parameter settings, the layout diffusion model can gradually recover the semantic layout graph that conforms to the joint conditional vector constraint from pure noise, providing spatial structure guidance for the second stage of high-definition image generation.

[0193] Furthermore, in the second stage of step S3 above, the high-definition diffusion model uses the semantic layout graph and joint condition vector as guiding conditions in the following specific way:

[0194] Obtain the semantic layout map generated in the first stage. The spatial resolution of the semantic layout map is higher than the spatial resolution of the latent code inside the high-definition diffusion model. Set the spatial resolution of the latent code of the high-definition diffusion model to 64×64 pixels and the number of channels to 4.

[0195] The semantic layout map is downsampled from the original resolution of 256×256 pixels to 64×64 pixels through bilinear interpolation to obtain the downsampled semantic layout map, which has 3 channels, corresponding to the red, green and blue color channels respectively.

[0196] The downsampled semantic layout graph and the joint condition vector C are combined. con The components are stitched together along the channel dimension.

[0197] Joint condition vector C con The dimension is 256, which is a one-dimensional vector. The downsampled semantic layout graph is a three-dimensional tensor with a size of 64×64×3.

[0198] To achieve concatenation along the channel dimension, the joint condition vector C is first... con The feature map is expanded to the same size as the downsampled semantic layout graph space by a fully connected layer, which maps the 256-dimensional vector to a 64×64×128 feature map.

[0199] The feature map is concatenated with the downsampled semantic layout map in the channel dimension to obtain a 64×64×131 three-dimensional tensor, where 3 comes from the semantic layout map and 128 comes from the conditional feature map, for a total of 131 channels;

[0200] Alternatively, another approach can be used, where the joint condition vector C con The feature map is copied and expanded to the same spatial size as the downsampled semantic layout map, that is, the same 256-dimensional vector is repeated at each spatial location to form a 64×64×256 feature map, which is then concatenated with the 64×64×3 semantic layout map to obtain a 64×64×259 three-dimensional tensor. Both methods can achieve channel dimension concatenation.

[0201] The stitched 3D tensor is used as an additional input to the noise predictor of the high-definition diffusion model;

[0202] The noise predictor of the high-definition diffusion model adopts a U-Net structure similar to that of the layout diffusion model, but the input layer is modified to accept the above-mentioned spliced ​​multi-channel input;

[0203] The input convolutional layers of U-Net map the number of channels in the input tensor to the number of channels in the internal features;

[0204] In each denoising iteration, the noise predictor simultaneously receives the latent code of the current step, the time step information, the joint condition vector, and the above-mentioned concatenated guiding tensor. The joint condition vector is injected through a cross-attention mechanism, and the concatenated guiding tensor is directly used as the input channel and concatenated with the current latent code in the channel dimension before entering the first convolutional layer.

[0205] In this way, the high-definition diffusion model is simultaneously constrained by the spatial structure of the semantic layout graph and guided by the global semantics of the joint conditional vector during the denoising process, generating high-definition brand marketing images that not only meet user preferences and emotions but also have a reasonable spatial layout.

[0206] Specifically, by using the aforementioned downsampling, channel stitching, and additional input methods, the semantic layout graph and joint conditional vector were used to guide the high-resolution diffusion model, ensuring the structural consistency of the final generated image.

[0207] Furthermore, the objective function for multi-objective reinforcement learning in step S4 is:

[0208]

[0209] R emotion This is an instantaneous emotional reward, calculated based on the real-time change in the user's emotional state after the generated high-definition brand marketing image is displayed.

[0210] Before image projection, the user's current emotional state triplet, including pleasure, arousal, and dominance, is obtained through the multimodal temporal convolutional network in step S2, denoted as V. before A before D before ;

[0211] After the image is projected, the same multimodal temporal convolutional network is used again to obtain the user's new emotional state triples, denoted as V. after A after D after The change in pleasure level is:

[0212]

[0213] The change in wakefulness is:

[0214]

[0215] The change in dominance is:

[0216]

[0217] Instantaneous emotional reward R emotion Take the weighted sum of the three changes mentioned above:

[0218]

[0219] Among them, w v w a w d The preset weighting coefficients are 0.5, 0.3, and 0.2 respectively.

[0220] When user emotions change in a positive direction, R emotion When R is positive; when it changes in a negative direction, emotion It is a negative value;

[0221] R brand As a long-term brand value reward, the reward value is obtained by weighting the user's repurchase probability and recommendation intention in a future preset period through a pre-trained user lifetime value prediction model.

[0222] The user lifetime value prediction model employs a deep neural network structure. Its inputs are the feature vectors of the user's historical interaction data and the embedding features of the currently generated high-definition brand marketing image. The output consists of two predicted values:

[0223] Probability of repurchase within the next 30 days, P repeat And the recommendation intention score P recommend Both values ​​range from 0 to 1;

[0224] Long-term brand value reward R brand The calculation formula is:

[0225]

[0226] The model is trained offline using user history behavior data accumulated on the platform. The training labels are whether the user made a repeat purchase and whether they made a recommendation in the next 30 days.

[0227] R risk This is a penalty for policy risk. The penalty value is provided by the policy risk score output by the hypergraph convolutional network.

[0228] In the hypergraph convolutional network of step S2, for the hyperedge corresponding to the currently generated image, the feature distance between the emotion node and the brand attribute node connected by the hyperedge is calculated, and the feature distance adopts the Euclidean distance metric d.

[0229] Preset safe distance threshold D safe The value is 2.0 when d≤D safe At that time, R risk The value is 0;

[0230] When d>D safe At that time, R risk Values , where D max The maximum penalty distance threshold is set to 5.0. This design applies a positive penalty when the semantic deviation between the emotion corresponding to the generated content and the brand attribute exceeds the safe range, guiding the reinforcement learning algorithm to avoid high-risk strategies.

[0231] α, β, and γ are preset weighting coefficients used to adjust the relative importance of instantaneous emotional rewards, long-term brand value rewards, and strategic risk penalties in the overall objective function, respectively.

[0232] The values ​​of α, β, and γ are dynamically configured according to the target focus of the marketing scenario. In scenarios that focus on short-term sentiment optimization, α is 0.6, β is 0.3, and γ is 0.1.

[0233] In scenarios that emphasize long-term brand building, α is set to 0.2, β to 0.7, and γ to 0.1.

[0234] In scenarios where strict control of strategy risk is required, α is set to 0.4, β to 0.3, γ to 0.3, and the sum of the three weighting coefficients equals 1.

[0235] Specifically, through the aforementioned objective function, the reinforcement learning agent calculates the total reward R corresponding to the current policy at each decision step, and uses this as the basis for updating the policy network parameters, so that the agent can gradually learn the generative control parameters that can achieve a balance between the user's instantaneous emotional satisfaction and long-term brand value.

[0236] Furthermore, in the aforementioned hypergraph convolutional network, the generation of the policy risk score is performed according to the following steps;

[0237] For each candidate generation strategy, there is a specific hyperedge in the hypergraph convolutional network. The hyperedge connects at least two of the three types of nodes, which must include emotion nodes and brand attribute nodes. Emotion nodes are represented by the real-time emotion node embedding vector extracted in step S2, and brand attribute nodes are represented by the predefined fixed coordinate embedding vector of the brand in the consumer preference semantic space.

[0238] During the operation of the hypergraph convolutional network, for each candidate generation strategy corresponding to the hyperedge, the embedding vectors of the emotion nodes and the brand attribute nodes connected by the hyperedge are extracted, and the feature distance between the two embedding vectors is calculated.

[0239] The feature distance uses Euclidean distance as a metric, and the calculation formula is as follows:

[0240] The feature distance is equal to the square root of the sum of the squares of the differences between the emotion node embedding vector and the brand attribute node embedding vector across all dimensions. Let the emotion node embedding vector be E with dimension d, and the brand attribute node embedding vector be B with the same dimension. Then the feature distance is:

[0241]

[0242] Preset feature distance threshold D threshold The threshold is determined during the model training phase based on the distribution range of brand attribute nodes, and is set to 1.5 times the average distance between all brand attribute nodes, for example, 2.0. The baseline value is defined as the benchmark value of the strategy risk score, and is set to 0.

[0243] When the calculated feature distance is less than or equal to the preset threshold D threshold If the candidate generation strategy is deemed to have no significant risk, the strategy risk score output baseline value is 0.

[0244] When the feature distance exceeds the preset threshold D threshold If the candidate generation strategy is deemed to have a risk, a strategy risk score higher than the baseline value is output.

[0245] The strategy risk score above the baseline value is calculated as follows:

[0246]

[0247] Among them, D max The maximum risk distance is set to 5.0, D. max The risk score S is determined based on the maximum Euclidean distance between all brand attribute nodes. In this embodiment, the maximum distance between brand nodes in the training set is 5.0, so we take 5.0. This formula makes the risk score S change continuously between 0 and 1. The larger the feature distance, the higher the risk score.

[0248] Specifically, the final output policy risk score is used in R in the multi-objective reinforcement learning objective function in step S4. risk In this way, the hypergraph convolutional network can quantify the semantic deviation between the emotion and brand attributes corresponding to each candidate generation strategy, and generate a strategy risk score that matches the risk level.

[0249] Furthermore, in step S4 above, the specific method for iteratively optimizing the generation control parameters of the cascade diffusion model is as follows:

[0250] The policy network was trained using the Proximal Policy Optimization (PPO) algorithm. This policy network adopted a multilayer perceptron structure, containing three fully connected hidden layers with 256, 128, and 64 neurons in each layer, respectively. The activation function used was the modified linear unit.

[0251] The input to the policy network is the joint condition vector C. con The vector has a dimension of 256 and is formed by concatenating the updated user preference node embedding vector and the sentiment node embedding vector. The output of the policy network consists of two parts.

[0252] The first part is the probability distribution of layout selection in the first stage of the cascade diffusion model;

[0253] The layout selection space includes six predefined macro layout types: center-focused, top-bottom split, left-right split, diagonal split, wraparound, and scattered.

[0254] The policy network outputs a six-dimensional vector, which is normalized by the softmax function to obtain the probability distribution of each layout type;

[0255] The second part consists of local detail control parameters in the second stage of the cascade diffusion model, which includes four continuous parameters: color saturation adjustment coefficient, contrast adjustment coefficient, product subject scaling ratio, and background blur degree. The value range of each parameter is 0 to 1.

[0256] The policy network outputs a four-dimensional vector, with each dimension corresponding to a control parameter, which is then mapped to the 0-1 interval by the sigmoid function.

[0257] In each strategy decision, the first step is to start from the joint condition vector C. con A vector is sampled from the distribution and input into the policy network to obtain the probability distribution of layout selection. A layout type is then sampled from this distribution.

[0258] Simultaneously, the specific values ​​of four local detail control parameters are output, and the sampled layout type and control parameters are applied to the cascade diffusion model to generate a semantic layout diagram and a high-definition brand marketing image.

[0259] By obtaining reward signals R through user interaction, the generated high-definition brand marketing images are displayed to users, and multimodal feedback data of users is collected, including facial expressions, voice tone, physiological signals and interactive behaviors such as clicks and dwell time.

[0260] Based on these feedback data, the reward signal R is calculated according to the multi-objective reinforcement learning objective function defined in step S4 above. The value of R is the reward corresponding to the current policy decision.

[0261] The PPO algorithm is used to update the policy network parameters. The PPO algorithm limits the step size of each update by using the probability ratio of pruning to avoid drastic policy changes. Let the old policy network parameters be θ. old The network parameters for the new strategy are θ;

[0262] For the trajectory data obtained from each interaction, calculate the estimated value A of the dominance function. t ;

[0263] The loss function of PPO consists of three parts: policy loss, value loss, and entropy regularization term. The formula for calculating policy loss is as follows:

[0264]

[0265] Where r t (θ) represents the probability ratio between the current policy and the old policy, and δ is the pruning hyperparameter with a value of 0.2;

[0266] The value loss adopts mean squared error loss, updates the value network used to estimate state value, and the entropy regularization term encourages policy exploration.

[0267] The total loss is the policy loss plus the value loss coefficient multiplied by the value loss, minus the entropy regularization coefficient multiplied by the entropy. The Adam optimizer is used for gradient descent updates, and the learning rate is set to 0.0003.

[0268] The given constraints applied during the update process include that the probability distribution entropy of the layout selection is not less than the preset minimum exploration threshold of 0.5, ensuring that the policy network continues to explore different layout types;

[0269] The update step size of local detail adjustment parameters is limited to no more than 0.1 per update to avoid drastic changes in generated content that could affect user experience;

[0270] Through the above method, the policy network parameters are continuously updated, causing the value of the weighted reward function R to gradually increase and converge to near the optimal value;

[0271] Specifically, the policy network trained using the PPO algorithm described above is able to determine the user's joint condition vector C at each step. con By dynamically selecting the optimal layout type and control parameters, a closed-loop iterative optimization of the control parameters for the cascade diffusion model is achieved.

[0272] Furthermore, after step S4 above, this method also includes a closed-loop feedback update step;

[0273] The high-definition brand marketing image generated in step S4 is displayed to the user, and new multimodal data is generated after the user views the image;

[0274] The new multimodal data includes voice signals, text data, facial expression video streams and physiological signals when users view images, as well as user interaction data on the images, including clicks, swipes, dwell time, sharing or favorites. This new multimodal data is collected as feedback data.

[0275] The feedback data is fed back to step S2. In step S2, the user preference node and real-time sentiment node in the heterogeneous hypergraph are updated using the feedback data.

[0276] The update method for user preference nodes is as follows: interactive behavior data and text comment data in the feedback data are used as new historical interaction items. The coordinates of the products or brands corresponding to these interaction items in the semantic space of consumer preferences are extracted. These new coordinates are merged with the coordinates of the original user historical interaction items. The distribution center is recalculated to obtain the updated long-term consumer preference embedding vector. The original user preference node embedding vector is replaced with this new vector.

[0277] The real-time emotion node is updated as follows: the voice signal, facial expression video stream and physiological signal in the feedback data are aligned in time, input into a multimodal temporal convolutional network, output a new emotion state triplet, and convert the triplet into a new emotion node embedding vector to replace the original real-time emotion node embedding vector. The brand attribute node remains unchanged.

[0278] After updating the heterogeneous hypergraph, repeat steps S2 to S4.

[0279] When step S2 is re-executed, the hypergraph convolution operation is performed again based on the updated user preference nodes and real-time sentiment nodes to generate a new joint conditional vector.

[0280] When step S3 is re-executed, the new joint conditional vector is input into the cascade diffusion model to generate a new high-definition brand marketing image;

[0281] When step S4 is re-executed, the reward signal is calculated based on the new user interaction feedback, and the generation control parameters of the cascade diffusion model are further optimized.

[0282] Specifically, through the above method, after each round of execution from step S1 to step S4 and obtaining user feedback, the system automatically triggers a new round of closed-loop feedback update, uses the feedback data to update the node representation in the heterogeneous hypergraph, and re-executes the subsequent steps on this basis, forming a continuous iterative optimization loop. This loop makes the user preference nodes and real-time emotion nodes continuously refined with each user interaction, and the joint conditional vector increasingly accurately reflects the user's consumption preferences and emotional state. The quality and personalization of the generated high-definition brand marketing images are continuously improved in the iteration.

[0283] Although alternative embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make further changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0284] The above specific embodiments further illustrate the purpose, technical solution and beneficial effects of this application. It should be understood that the above are only specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of this application should be included within the scope of protection of this invention.

Claims

1. A user consumption preference multi-modal representation learning and content generation method for brand emotional marketing, characterized in that, Includes the following steps: S1, real-time collection of user's multimodal data through multimodal perception components deployed on terminal devices, the multimodal data including voice signals, text data, facial expression video streams and physiological signals; S2, Construct a heterogeneous hypergraph, wherein the node types of the heterogeneous hypergraph include: user preference nodes generated by mapping user historical interaction data in the consumption preference semantic space, real-time emotion nodes extracted by the multimodal data through a multimodal temporal convolutional network, and predefined brand attribute nodes. The heterogeneous hypergraph is convolved by a hypergraph convolutional network to generate a joint conditional vector, which is formed by concatenating the updated user preference node embedding vector and the sentiment node embedding vector. S3, input the joint condition vector into the cascaded diffusion model, which includes a first stage and a second stage: In the first stage, the layout diffusion model generates a semantic layout map from pure noise through a multi-step denoising process, using the joint conditional vector as a condition. The semantic layout map describes the macroscopic composition of the image to be generated using different color blocks and contours. In the second stage, the high-definition diffusion model uses the semantic layout graph and the joint conditional vector as guiding conditions to generate high-definition brand marketing images through multi-step denoising. S4 employs a multi-objective reinforcement learning algorithm, using the weighted sum of user instantaneous emotional satisfaction and long-term brand value as the objective function, to iteratively optimize the generation control parameters of the cascade diffusion model.

2. The method of claim 1, wherein, The specific method for constructing the user preference node in step S2 is as follows: Collect users' historical interaction data on shopping platforms, including purchase records, browsing history, favorites behavior, and text comments; Each product or brand is mapped to a continuous consumer preference semantic space by a pre-trained variational autoencoder, which includes design style dimension, functional value dimension, brand value dimension and emotional association dimension. The distribution center of the user's historical interaction items in the semantic space of consumption preferences is used as the long-term consumption preference embedding vector of the user, and the embedding vector is added to the heterogeneous hypergraph as the user preference node.

3. The method according to claim 1, characterized in that, The specific method for extracting real-time sentiment nodes in step S2 is as follows: The multimodal data is aligned in time and input into the multimodal temporal convolutional network; The multimodal temporal convolutional network outputs a triplet representing the user's current emotional state, which includes pleasure, arousal, and dominance. The triplet is then converted into an emotional node embedding vector and added to a heterogeneous hypergraph.

4. The method according to claim 1, characterized in that, The process of the hypergraph convolutional network performing convolution operations on heterogeneous hypergraphs includes: For each hyperedge, the embedding vectors of all nodes connected by the hyperedge are aggregated using mean pooling or attention mechanism to generate the hyperedge feature vector. For each node, collect the hyperedge feature vectors containing all hyperedges of that node, aggregate them again, and update the embedding representation of that node with a learnable weight matrix and bias parameters.

5. The method according to claim 1, characterized in that, The process of generating the semantic layout graph using the layout diffusion model in step S3 employs the following denoising iterative formula: ； Where, x t Let α be the noise latent code at step t, where t is the current time step. t For noise scheduling parameters, For cumulative product, ε θ For a noise predictor parameterized by U-Net or Diffusion Transformer, Ccon is the joint conditional vector, σ t Let ε be the standard deviation of the Gaussian noise, and ε be the random noise sampled from the standard normal distribution.

6. The method according to claim 1, characterized in that, The specific method by which the high-definition diffusion model described in step S3 uses both the semantic layout graph and the joint condition vector as guiding conditions is as follows: The semantic layout graph is downsampled to the same spatial resolution as the latent code of the high-definition diffusion model, and the downsampled semantic layout graph is concatenated with the joint conditional vector in the channel dimension as an additional input to the noise predictor of the high-definition diffusion model.

7. The method according to claim 1, characterized in that, The objective function for multi-objective reinforcement learning in step S4 is: ； wherein R emotion is the instantaneous emotional reward, calculated according to the change in the user's real-time emotional state after the generated high-definition brand marketing image is posted; R brand For long-term brand value rewards, the user's repurchase probability and recommendation willingness weighted in the future preset period are estimated by a pre-trained user life cycle value prediction model. R risk For the strategy risk penalty, the strategy risk score output by the hypergraph convolution network is provided, and α, β, and γ are preset weight coefficients.

8. The method according to claim 7, characterized in that, The strategy risk score is generated as follows: In the hypergraph convolutional network, for each candidate generation strategy corresponding to the hyperedge, the feature distance between the emotion node connected by the hyperedge and the brand attribute node is calculated. When the feature distance exceeds a preset threshold, a strategy risk score higher than the baseline value is output.

9. The method according to claim 1, characterized in that, The specific method for iteratively optimizing the generation control parameters of the cascade diffusion model in step S4 is as follows: A policy network is trained using a PPO algorithm, the policy network taking as input the joint condition vector C con outputs a probability distribution over layout choices in the first stage of the cascading diffusion model and local detail regulation parameters in the second stage. Feedback data is obtained through user interaction. Based on the feedback data and the objective function, the total reward value R is calculated and the policy network parameters are updated to maximize the total reward value R under given constraints.

10. The method according to any one of claims 1 to 9, characterized in that, It also includes a closed-loop feedback update step: The new multimodal data generated after the high-definition brand marketing image generated in step S4 interacts with the user is fed back to step S2 to update the user preference nodes and real-time emotion nodes in the heterogeneous hypergraph, and steps S2 to S4 are re-executed to form a continuous iterative optimization loop.

Images