An advertising marketing recommendation method based on deep reinforcement learning

Abstract

The present invention discloses an advertising marketing recommendation method based on deep reinforcement learning, which relates to the technical field of computer data processing, including integrating user behavior data, context information and advertising features, and processing them to form a multimodal feature vector; performing feature fusion on the multimodal feature vector to obtain a fused user comprehensive feature vector; dynamically updating the user portrait using an online learning algorithm according to the user comprehensive feature vector; designing an intelligent body based on deep reinforcement learning according to the user portrait and context information, and formulating an advertising recommendation strategy; collecting new data generated when the user interacts with the recommended advertisement, and optimizing the advertising recommendation strategy; the present invention not only improves the click-through rate and conversion rate of advertisements by dynamically updating user portraits and optimizing the advertising recommendation strategy, but also provides advertisers with a higher return on investment, thereby gaining an advantage in a fiercely competitive market.

Description

Technical Field

[0001] The present invention relates to the field of computer data processing technology, and in particular to an advertising marketing recommendation method based on deep reinforcement learning. Background Art

[0002] In recent years, advertising recommendation systems have been widely used in the field of personalized marketing, especially the combination of deep learning and reinforcement learning, which has significantly improved the accuracy of advertising recommendations. Traditional advertising recommendation systems mostly rely on static user features and historical data, and often cannot effectively respond to the rapid changes in user behavior and diversified needs. However, with the increasing diversity of user behavior data, contextual information, and advertising features, how to integrate this information and achieve dynamic updates has become a technical problem that needs to be solved urgently.

[0003] Existing technologies have deficiencies in real-time updating and feature fusion of user features. Many systems cannot efficiently process multimodal data, resulting in poor recommendation results. For example, a single feature extraction method often cannot fully capture changes in user interests, while static user portraits cannot adapt to rapid changes in user behavior. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides an advertising marketing recommendation method based on deep reinforcement learning to solve the problem of how to dynamically update user portraits and optimize advertising recommendation strategies.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, an embodiment of the present invention provides an advertising marketing recommendation method based on deep reinforcement learning, which includes integrating user behavior data, context information and advertising features, and processing them to form a multimodal feature vector;

[0008] Perform feature fusion on the multimodal feature vectors to obtain the fused user comprehensive feature vector;

[0009] Based on the user's comprehensive feature vector, the user profile is dynamically updated using an online learning algorithm;

[0010] Design an intelligent agent based on deep reinforcement learning and formulate an advertising recommendation strategy based on user portraits and context information;

[0011] Collect new data generated when users interact with recommended ads and optimize ad recommendation strategies.

[0012] As a preferred solution of the advertising marketing recommendation method based on deep reinforcement learning described in the present invention, wherein: the user behavior data includes click records, search history and purchase records;

[0013] The context information includes geographic location, weather conditions and timestamp;

[0014] The advertisement features include the category, keywords, images and videos of the advertisement.

[0015] As a preferred solution of the advertising marketing recommendation method based on deep reinforcement learning described in the present invention, the user behavior data, context information and advertising features are processed to form a multimodal feature vector, including the following steps:

[0016] For user behavior data, we use the Jieba word segmentation tool for Chinese word segmentation, the NLTK tool for English word segmentation, and the BERT model for word embedding to convert the segmented text into a fixed-length text feature vector.

[0017] For the image in the advertisement feature, input the image into the ResNet-50 model, extract the feature vector before the last fully connected layer, and normalize the extracted feature vector to obtain a standardized image feature vector;

[0018] For the video in the advertisement feature, the video is input into the 3D-CNN model frame by frame, the feature vector of each frame is extracted, and the feature vector of each frame is average pooled to obtain the video feature vector;

[0019] For context information, use geocoding services to convert geographic locations into longitude and latitude coordinates, standardize timestamp data and convert them into hours of the day, and use one-hot encoding to encode weather data to represent different weather types.

[0020] The text feature vector, image feature vector, video feature vector and context information are concatenated together to form a multimodal feature vector, which is expressed as:

[0021] v=[v text , V image , V video ,t,V weather ];

[0022] Among them, v is a multimodal feature, v text is the text feature vector, V image is the image feature vector, V video is the video feature vector, t is the timestamp, V weather The weather type represented by one-hot encoding.

[0023] As a preferred solution of the advertising marketing recommendation method based on deep reinforcement learning described in the present invention, wherein: performing feature fusion on multimodal feature vectors to obtain a fused user comprehensive feature vector includes the following steps:

[0024] Design a multi-layer perceptron, each hidden layer uses the ReLU activation function, the last layer uses the linear activation function, and the first layer accepts the multimodal feature vector as input. The calculation expression of the hidden layer is,

[0025] h l =σ(W l h l-1 +b l );

[0026] Among them, h l is the output vector of the lth layer, W l is the weight matrix of the lth layer, h l-1 is the output vector of the l-1th layer, σ is the activation function, b l is the bias vector of the lth layer;

[0027] The output layer outputs the fused user comprehensive feature vector, expressed as,

[0028] V fina1 =W L h L-1 +b L ;

[0029] Among them, v final is the user comprehensive feature vector, b L is the bias vector of the last layer, W L is the weight matrix of the last layer, h L -1 is the output vector of the second-to-last layer.

[0030] As a preferred solution of the advertising marketing recommendation method based on deep reinforcement learning described in the present invention, the following steps are included: dynamically updating the user portrait using an online learning algorithm according to the user's comprehensive feature vector:

[0031] Generate an initial user profile based on the information provided by the user during registration and past behavior data;

[0032] According to the user's comprehensive feature vector, the online gradient descent learning algorithm is used to dynamically update the user portrait, which is expressed as:

[0033]

[0034] Among them, θ t+1 is the parameter of the updated user profile, θ t is the parameter of the current user portrait, η is the learning rate, is the gradient of the loss function and y is the target label.

[0035] As a preferred solution of the advertising marketing recommendation method based on deep reinforcement learning described in the present invention, wherein: according to the user portrait and context information, an intelligent agent based on deep reinforcement learning is designed, including the following steps:

[0036] Define the state space based on user portraits and context information, set the action space to represent the agent's selection of recommended ads, define the reward mechanism based on the interaction results between the user and the recommended ads, and select the deep Q network algorithm as the deep reinforcement learning algorithm framework;

[0037] The agent estimates the Q value of the state-action pair using a deep neural network and updates the Q value according to the Bellman equation, expressed as,

[0038]

[0039] Where Q(s, a) is the expected return of taking action a in state s, α is the learning rate for updating the Q value, R(s, a) is the reward immediately obtained after taking action a in state s, γ is the discount factor, and s is the next state after taking action a. is the maximum Q value of all actions a in the next state s;

[0040] The experience replay mechanism is used to store the historical state, action, reward and next state of the intelligent agent, and the intelligent agent is trained to obtain a trained intelligent agent based on deep reinforcement learning.

[0041] As a preferred solution of the advertising marketing recommendation method based on deep reinforcement learning described in the present invention, formulating an advertising recommendation strategy refers to using the designed intelligent agent to select the action with the maximum Q value as the recommended advertisement according to the current state, and generating an advertising recommendation strategy according to the continuous updating of the Q value.

[0042] As a preferred solution of the advertising marketing recommendation method based on deep reinforcement learning described in the present invention, the new data generated when users interact with recommended advertisements and the advertising recommendation strategy are collected, including the following steps:

[0043] Collect new data generated when users interact with recommended ads, format the collected data into a standard format, store the new data in the experience replay buffer, randomly extract a batch of data pairs from the experience replay buffer, use the randomly sampled data pairs to update the Q value, dynamically adjust the reward structure based on user feedback, apply the new reward structure to the deep reinforcement learning model, observe the learning process and recommendation effect of the intelligent agent, and continuously adjust and optimize the reward structure based on monitoring results and user feedback to ensure that the intelligent agent can adapt to changes in user behavior, thereby optimizing the advertising recommendation strategy.

[0044] In a second aspect, an embodiment of the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the advertising marketing recommendation method based on deep reinforcement learning as described in the first aspect of the present invention is implemented.

[0045] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the advertising marketing recommendation method based on deep reinforcement learning as described in the first aspect of the present invention.

[0046] The beneficial effects of the present invention are as follows: the construction of multimodal feature vectors enables the system to comprehensively consider the user's behavioral data, contextual information and advertising features, thereby forming a more comprehensive user portrait. The introduction of online learning algorithms ensures the real-time update of user portraits, enabling the system to quickly respond to changes in user behavior and optimize advertising recommendation strategies. This dynamic adjustment mechanism effectively avoids the recommendation lag problem caused by static features in traditional systems. In addition, the intelligent agent combined with deep reinforcement learning can dynamically adjust the reward structure by continuously learning user feedback, thereby further improving the effectiveness of recommendations. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0048] Figure 1 This is a flowchart of the advertising and marketing recommendation method based on deep reinforcement learning in Example 1.

[0049] Figure 2 This is a flow chart of dynamically updating the advertisement recommendation strategy in Example 1. DETAILED DESCRIPTION

[0050] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0051] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0052] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0053] Example 1, reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, and provides an advertising marketing recommendation method based on deep reinforcement learning, comprising the following steps:

[0054] S1. Integrate user behavior data, context information and advertising features, and process them to form a multimodal feature vector, including the following steps:

[0055] User behavior data includes click records, search history, and purchase records; context information includes geographic location, weather conditions, and timestamps; and ad features include ad categories, keywords, images, and videos;

[0056] Use hash tables to store data entries for user behavior data, context information, and advertising features, remove duplicates by checking hash values, use anomaly detection algorithms (such as statistical-based methods) to identify and delete outliers, and use mean values ​​to fill in missing numeric data and mode values ​​to fill in missing categorical data;

[0057] For user behavior data, we use the jieba (jieba word segmentation) tool for Chinese word segmentation, the NLTK (Natural Language Toolkit) tool for English word segmentation, and the BERT (Bidirectional Encoder Representations from Transformers) model for word embedding. The segmented text is converted into a fixed-length text feature vector, expressed as:

[0058]

[0059] Among them, v text is the text feature vector, v i is the embedding vector corresponding to the i-th word after word segmentation, n is the number of words in the text, and i is the index of the word;

[0060] For the image in the advertisement feature, the image is input into the ResNet-50 (50-layer Residual Network) model, the feature vector before the last fully connected layer is extracted, and the extracted feature vector is normalized to obtain the standardized image feature vector, which is expressed as,

[0061]

[0062] Among them, v image is the image feature vector, v raw is the feature vector before the last fully connected layer extracted from the ResNet-50 model, ||v raw ||2 is the vector v raw The Euclidean norm of ;

[0063] For the video in the advertisement feature, the video is input into the 3D-CNN (Three-Dimensional Convolutional Neural Networks) model frame by frame, the feature vector of each frame is extracted, and the feature vector of each frame is averaged and pooled to obtain the video feature vector, which is expressed as:

[0064]

[0065] Among them, v video is the video feature vector, v f is the feature vector of the f-th frame in the video, F is the number of frames in the video, and f is the index of the frame;

[0066] For context information, use the geocoding service to convert the geographic location into longitude and latitude coordinates, standardize the timestamp data and convert it into hours in a day, and use one-hot encoding to encode the weather data to represent different weather types. Assuming that there are four types of weather: sunny, cloudy, rainy, and snowy, the one-hot encoding is represented as [1,0,0,0] to represent a sunny day, and so on;

[0067] The text feature vector, image feature vector, video feature vector, and context information (such as timestamp and weather type) are concatenated together to form a multimodal feature vector, expressed as,

[0068] v=[v text ,v image ,v video ,t,v weather ];

[0069] Among them, v is a multimodal feature, v text is the text feature vector, v image is the image feature vector, v video is the video feature vector, t is the timestamp, indicating the number of hours in a day, and v weather The weather type represented by one-hot encoding.

[0070] It should be noted that by processing user behavior data, contextual information, and advertising features to form a multimodal feature vector, it is possible to capture user information needs from multiple angles and improve the richness and diversity of feature vectors. First, through hash table deduplication and anomaly detection, the quality of the data is guaranteed and the impact of noise data on subsequent analysis is avoided. Secondly, word segmentation tools and pre-trained models (such as BERT, ResNet-50, and 3D-CNN) are used to process text, images, and videos, converting unstructured multimedia data into structured feature vectors, thereby improving the processability of the data. Finally, by splicing multimodal feature vectors, a comprehensive user feature representation is formed, laying the foundation for subsequent feature fusion and user portrait updates.

[0071] S2, feature fusion is performed on the multimodal feature vector to obtain the fused user comprehensive feature vector, including the following steps:

[0072] Design a multi-layer perceptron (MLP). The network contains several hidden layers. Each hidden layer uses the ReLU (rectified linear unit) activation function. The last layer uses a linear activation function. The first layer accepts a multimodal feature vector as input. The calculation expression of the hidden layer is:

[0073] h l =σ(W l h l-1 +b l );

[0074] Among them, h l is the output vector of the lth layer, indicating the activation state of the lth layer neural network, which is used to pass to the next layer as input, W l is the weight matrix of the lth layer, which is used to calculate the linear combination of the lth layer. The output of the previous layer is transformed into the input of the current layer through matrix multiplication. l-1 is the output vector of the l-1th layer, indicating the activation state of the l-1th layer neural network, as the input of the lth layer, σ is the activation function, which introduces nonlinear characteristics into the network so that the network can fit complex function mapping, b l is the bias vector of the lth layer;

[0075] The output layer outputs the fused user comprehensive feature vector, expressed as,

[0076] V final =W L h L-1 +b L ;

[0077] Among them, v finalis the user comprehensive feature vector, which represents the final feature vector after multi-layer perceptron processing and is used for subsequent recommendation strategy formulation. L is the bias vector of the last layer. Adding a bias term to the output of the last layer allows the network to move the position of the activation function to better fit the data. L is the weight matrix of the last layer, which is used to calculate the linear combination of the last layer and transform the output of the previous layer into the final feature vector. L -1 is the output vector of the penultimate layer, indicating the activation state of the penultimate neural network, which serves as the input of the last layer;

[0078] Assuming that the MLP contains two hidden layers, each with a dimension of 512 and an output layer with a dimension of 256, the calculation process of the MLP is as follows,

[0079] The input layer is denoted as V multi →h0;

[0080] Hidden layer 1 is represented by h1=ReLU(W1h0+b1);

[0081] Hidden layer 2 is represented by h2=ReLU(W2h1+b2);

[0082] The output layer is denoted as V final =W3h2+b3;

[0083] The final fused user comprehensive feature vector V final The dimension is 256.

[0084] It should be noted that by designing a multi-layer perceptron (MLP) for feature fusion, it is not only possible to extract important information from each modal feature, but also to discover the correlation between different modal features, thereby enhancing the expressiveness of the features. The nonlinear activation function (such as ReLU) in the multi-layer perceptron introduces nonlinear characteristics, enabling the network to learn more complex feature combinations, improving the generalization ability and prediction accuracy of the model. The output layer uses a linear activation function to retain the true value of the feature and avoid information loss caused by nonlinear functions. The final output user comprehensive feature vector provides more accurate data support for subsequent user portrait updates and recommendation strategy formulation.

[0085] S3. Based on the user's comprehensive feature vector, the user profile is dynamically updated using an online learning algorithm, including the following steps:

[0086] Generate an initial user profile based on the information provided by the user during registration and past behavior data. The user profile includes static attributes (such as age, gender, registration date) and dynamic features (such as recent behavior characteristics and interest preferences);

[0087] According to the user's comprehensive feature vector, the online gradient descent learning algorithm is used to dynamically update the user portrait, which is expressed as:

[0088]

[0089] Among them, θ t+1 is the parameter of the updated user profile, θ t is the parameter of the current user portrait, η is the learning rate, which controls the step size of each update, is the gradient of the loss function, relative to the current parameter θ t The partial derivative of , based on the current eigenvector v final and the target label y, where y is the target label.

[0090] It should be noted that by updating the user profile in real time through the online gradient descent algorithm, it is possible to quickly respond to changes in user behavior and maintain the timeliness of the user profile. This method not only improves the accuracy and personalization of the user profile, but also can adjust the recommendation strategy in time when the user behavior pattern changes, enhancing the flexibility and adaptability of the recommendation system. By continuously learning new user behaviors, the system can better meet the personalized needs of users.

[0091] S4. Based on user portraits and context information, design an intelligent agent based on deep reinforcement learning and formulate an advertising recommendation strategy, including the following steps:

[0092] Define the state space based on user profile and context information. The state includes user profile features (such as interests, recent behaviors) and context information (such as geographic location, time, weather). Set the action space to represent the agent's selection of recommended ads (including electronic products, clothing, food, etc.), which can be represented by ad ID or ad category. Define the reward mechanism based on the interaction results between the user and the recommended ads. User stay time or purchase behavior can be used as a supplement to the reward. Select algorithms such as Deep Q Network (DQN) or Policy Gradient as the deep reinforcement learning algorithm framework;

[0093] The agent estimates the Q value of the state-action pair using a deep neural network and updates the Q value according to the Bellman equation, expressed as,

[0094]

[0095] Where Q(s, a) is the expected return of taking action a in state s, α is the learning rate for updating Q values, R(s, a) is the reward immediately obtained after taking action a in state s, γ is the discount factor used to balance the importance of immediate rewards and future rewards, and s′ is the next state after taking action a. is the maximum Q value of all actions a′ in the next state s′;

[0096] The experience replay mechanism is used to store the historical state, action, reward and next state of the intelligent agent, and the intelligent agent is trained to obtain a trained intelligent agent based on deep reinforcement learning.

[0097] Using the designed intelligent agent, according to the current state, the action with the maximum Q value is selected as the recommended advertisement, and the advertisement recommendation strategy is generated according to the continuous updating of the Q value.

[0098] It should be noted that by defining the state space, action space, and reward mechanism, and using deep reinforcement learning algorithms (such as DQN or policy gradient), the system can automatically learn the optimal recommendation strategy. This method can dynamically adjust the recommendation strategy based on real-time user feedback, improve the accuracy of recommendations and user satisfaction. Through the experience replay mechanism, the agent can learn from past experience, avoid the problem of over-reliance on real-time data, and enhance the stability and robustness of the system.

[0099] S5. Collect new data generated when users interact with recommended ads and optimize ad recommendation strategies, including the following steps:

[0100] Collect new data generated when users interact with recommended ads, including but not limited to users' clicks, dwell time, purchase behavior, etc., format the collected data into a standard format for subsequent processing, record whether the user clicks on the recommended ad, record the user's dwell time on the ad page, record whether the user makes a purchase after seeing the ad, store the new data in the experience replay buffer, randomly extract a batch of data pairs from the experience replay buffer, use the randomly sampled data pairs to update the Q value, dynamically adjust the reward structure based on user feedback, apply the new reward structure to the deep reinforcement learning model, observe the learning process and recommendation effect of the intelligent agent, and continuously adjust and optimize the reward structure based on monitoring results and user feedback to ensure that the intelligent agent can adapt to changes in user behavior, thereby optimizing the advertising recommendation strategy.

[0101] It should be noted that by collecting user feedback data and using it to optimize recommendation strategies, the system can continuously improve itself and enhance the quality of recommendations. The experience replay buffer enables the system to learn from historical data, avoiding the problem of data sparsity. By updating the Q value through random sampling, the system can balance the relationship between exploration and utilization, trying new recommendation strategies while maintaining the effectiveness of existing strategies. Dynamically adjusting the reward structure ensures that the agent can adapt to changes in user behavior patterns, improving the long-term performance and user experience of the recommendation system.

[0102] This embodiment also provides a computer device, which is suitable for the advertising marketing recommendation method based on deep reinforcement learning, including: a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the advertising marketing recommendation method based on deep reinforcement learning proposed in the above embodiment.

[0103] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0104] This embodiment also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the advertising marketing recommendation method based on deep reinforcement learning as proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, disk or optical disk.

[0105] In summary, the present invention constructs a multimodal feature vector so that the system can comprehensively consider the user's behavioral data, contextual information and advertising features to form a more comprehensive user portrait. The introduction of the online learning algorithm ensures the real-time update of the user portrait, enabling the system to quickly respond to changes in user behavior and optimize the advertising recommendation strategy. This dynamic adjustment mechanism effectively avoids the recommendation lag problem caused by static features in traditional systems. In addition, the intelligent agent combined with deep reinforcement learning can dynamically adjust the reward structure by continuously learning user feedback, thereby further improving the effectiveness of recommendations.

Claims

1. An advertising marketing recommendation method based on deep reinforcement learning, characterized in that: include, Integrate user behavior data, context information, and advertising features, and process them to form a multimodal feature vector; Perform feature fusion on the multimodal feature vectors to obtain the fused user comprehensive feature vector; Based on the user's comprehensive feature vector, the user profile is dynamically updated using an online learning algorithm; Design an intelligent agent based on deep reinforcement learning and formulate an advertising recommendation strategy based on user portraits and context information; Collect new data generated when users interact with recommended ads and optimize ad recommendation strategies; The user behavior data includes click records, search history and purchase records; The context information includes geographic location, weather conditions and timestamp; The advertisement features include the category, keywords, images and videos of the advertisement; Processing user behavior data, context information and advertising features to form a multimodal feature vector includes the following steps: For user behavior data, we use the Jieba word segmentation tool for Chinese word segmentation, the NLTK tool for English word segmentation, and the BERT model for word embedding to convert the segmented text into a fixed-length text feature vector. For the image in the advertisement feature, input the image into the ResNet-50 model, extract the feature vector before the last fully connected layer, and normalize the extracted feature vector to obtain a standardized image feature vector; For the video in the advertisement feature, the video is input into the 3D-CNN model frame by frame, the feature vector of each frame is extracted, and the feature vector of each frame is average pooled to obtain the video feature vector; For context information, use geocoding services to convert geographic locations into longitude and latitude coordinates, standardize timestamp data and convert them into hours of the day, and use one-hot encoding to encode weather data to represent different weather types. The text feature vector, image feature vector, video feature vector and context information are concatenated together to form a multimodal feature vector, which is expressed as: V=[v text ,v image ,v video ,t,v weather ]; Among them, v is a multimodal feature, v text is the text feature vector, v image is the image feature vector, v video is the video feature vector, t is the timestamp, v weather The weather type represented by one-hot encoding.

2. The advertising marketing recommendation method based on deep reinforcement learning according to claim 1, characterized in that: The multimodal feature vectors are subjected to feature fusion to obtain the fused user comprehensive feature vector, including the following steps: Design a multi-layer perceptron, each hidden layer uses the ReLU activation function, the last layer uses the linear activation function, and the first layer accepts the multimodal feature vector as input. The calculation expression of the hidden layer is, h l =σ(W l h l-1 +b l ); Among them, h l is the output vector of the lth layer, W l is the weight matrix of the lth layer, h l-1 is the output vector of the l-1th layer, σ is the activation function, b l is the bias vector of the lth layer; The output layer outputs the fused user comprehensive feature vector, expressed as, v final =W L h L-1 +b L ; Among them, v final is the user comprehensive feature vector, b L is the bias vector of the last layer, W L is the weight matrix of the last layer, h L-1 is the output vector of the second-to-last layer.

3. The advertising marketing recommendation method based on deep reinforcement learning according to claim 2, characterized in that: Based on the user's comprehensive feature vector, the user profile is dynamically updated using an online learning algorithm, including the following steps: Generate an initial user profile based on the information provided by the user during registration and past behavior data; According to the user's comprehensive feature vector, the online gradient descent learning algorithm is used to dynamically update the user portrait, which is expressed as: Among them, θ t+1 is the parameter of the updated user profile, θ t is the parameter of the current user portrait, η is the learning rate, is the gradient of the loss function and y is the target label.

4. The advertising marketing recommendation method based on deep reinforcement learning according to claim 3, characterized in that: According to the user profile and context information, design an intelligent agent based on deep reinforcement learning, including the following steps: Define the state space based on user portraits and context information, set the action space to represent the agent's selection of recommended ads, define the reward mechanism based on the interaction results between the user and the recommended ads, and select the deep Q network algorithm as the deep reinforcement learning algorithm framework; The agent estimates the Q value of the state-action pair using a deep neural network and updates the Q value according to the Bellman equation, expressed as, Where Q(s, a) is the expected return of taking action a in state s, α is the learning rate for updating Q values, R(s, a) is the reward immediately obtained after taking action a in state s, γ is the discount factor, and s′ is the next state after taking action a. is the maximum Q value of all actions a′ in the next state s′; The experience replay mechanism is used to store the historical state, action, reward and next state of the intelligent agent, and the intelligent agent is trained to obtain a trained intelligent agent based on deep reinforcement learning.

5. The advertising marketing recommendation method based on deep reinforcement learning according to claim 4, characterized in that: Formulating an advertising recommendation strategy means using the designed intelligent agent to select the action with the maximum Q value as the recommended advertisement based on the current state, and generating an advertising recommendation strategy based on the continuous updating of the Q value.

6. The advertising marketing recommendation method based on deep reinforcement learning according to claim 5, characterized in that: Collect new data generated when users interact with recommended ads and optimize ad recommendation strategies, including the following steps: Collect new data generated when users interact with recommended ads, format the collected data into a standard format, store the new data in the experience replay buffer, randomly extract a batch of data pairs from the experience replay buffer, use the randomly sampled data pairs to update the Q value, dynamically adjust the reward structure based on user feedback, apply the new reward structure to the deep reinforcement learning model, observe the learning process and recommendation effect of the intelligent agent, and continuously adjust and optimize the reward structure based on monitoring results and user feedback to ensure that the intelligent agent can adapt to changes in user behavior, thereby optimizing the advertising recommendation strategy.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the advertising marketing recommendation method based on deep reinforcement learning as described in any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the advertising marketing recommendation method based on deep reinforcement learning described in any one of claims 1 to 6 are implemented.

Images