Advertisement recommendation method and system based on deep reinforcement learning

Abstract

The invention provides an advertisement recommendation method and system based on deep reinforcement learning. The method comprises the following steps: receiving a real-time interaction signal of a user, analyzing time sequence characteristics in the real-time interaction signal to extract behavior characteristics of the user, and identifying a contextual model of the user; modeling the interest preference of the user to generate a personalized interest dynamic graph; performing intelligent exploration and decision processing on the optimal advertisement content and the corresponding pushing opportunity to obtain an advertisement recommendation strategy; calculating the optimal pushing opportunity corresponding to the optimal advertisement content from the optimal advertisement content and the corresponding pushing opportunity, and pushing the optimal advertisement content to the user based on the optimal pushing opportunity; establishing a real-time feedback mechanism to obtain an adjusted advertisement recommendation strategy; and performing comprehensive evaluation processing on the advertisement effect of the adjusted advertisement recommendation strategy to generate an advertisement effect evaluation result. According to the technical scheme provided by the invention, the flexibility and efficiency of advertisement recommendation can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of advertisement recommendation, and in particular, to an advertisement recommendation method and system based on deep reinforcement learning. Background Art

[0002] In the modern digital marketing environment, advertising recommendation systems play a vital role. With the popularization of the Internet and mobile devices, users are exposed to a large amount of information every day. How to accurately push advertisements that meet their interests and needs to users has become a major challenge.

[0003] At present, most advertising recommendation systems rely mainly on traditional machine learning methods or rule-based recommendation algorithms. These solutions usually include the following: analyzing the user's historical behavior data, finding similar user groups, and then recommending content that these users like to the target users; recommending relevant content based on the user's past behavior and preferences; setting a series of rules to match user characteristics and advertising content, but lacking flexibility and adaptability.

[0004] Although existing solutions can achieve advertising recommendations to a certain extent, there are still some significant shortcomings: traditional methods mostly make recommendations based on historical data, which makes it difficult to capture users' immediate interest changes and situational patterns, resulting in inaccurate recommendations; existing solutions usually ignore the choice of push timing when selecting advertising content, and cannot maximize the effectiveness of advertising and user engagement; many systems lack an effective real-time feedback mechanism and are unable to adjust recommendation strategies in a timely manner to adapt to changes in user behavior, affecting the long-term performance of the system and user experience. Summary of the invention

[0005] The embodiments of the present application provide an advertisement recommendation method and system based on deep reinforcement learning, so as to solve the problems of poor flexibility and low efficiency of advertisement recommendation in the prior art.

[0006] In a first aspect, an embodiment of the present application provides an advertisement recommendation method based on deep reinforcement learning, comprising: receiving a real-time interaction signal of a user, wherein the real-time interaction signal includes the browsing behavior, clicking behavior and dwell time of the user, analyzing the time series characteristics in the real-time interaction signal to extract the behavior characteristics of the user, and identifying the scenario mode of the user; According to the scenario mode and the behavior characteristics, a graph neural network combined with a node importance scoring mechanism and an attention mechanism is used to model the user's interest preferences to generate a personalized interest dynamic map, which is used to reflect the user's immediate interest trends, long-term interest trends, and potential demand points; Based on the personalized interest dynamic graph, a deep reinforcement learning algorithm combining dual Q learning and policy gradient is used, and the response probability distribution of the user at different push timings is simulated through a Markov decision process, and the optimal advertising content and the corresponding push timing are intelligently explored and decided to obtain an advertising recommendation strategy, which includes the optimal advertising content and the corresponding push timing; From the optimal advertising content and the corresponding push timing, the optimal push timing corresponding to the optimal advertising content is calculated, and the optimal advertising content is pushed to the user based on the optimal push timing; and a real-time feedback mechanism is established based on Bayesian optimization, and the user's feedback information is used to optimize the advertising recommendation strategy using an online learning framework to obtain an adjusted advertising recommendation strategy; A multi-dimensional effect evaluation system integrating hierarchical reinforcement learning and multi-objective evolutionary algorithm is applied to comprehensively evaluate the advertising effect of the adjusted advertising recommendation strategy according to traditional marketing goals and user experience indicators to generate advertising effect evaluation results.

[0007] Optionally, based on the personalized interest dynamic graph, a deep reinforcement learning algorithm combining dual Q learning and policy gradient is used, and the response probability distribution of the user at different push timings is simulated through a Markov decision process, and the optimal advertising content and the corresponding push timing are intelligently explored and decided to obtain an advertising recommendation strategy, including: Using the personalized interest dynamic graph, the user's interest preference and the user's scenario mode are analyzed and processed to obtain input data; based on the input data, a deep reinforcement learning algorithm combining dual Q learning and policy gradient is used to simulate and process a variety of advertising content and corresponding push timing combinations to obtain potential impact assessment results; According to the potential impact assessment result, using the Markov decision process to simulate the user's response probability distribution at different push timings, calculating the probability distribution of each type of the advertising content being accepted by the user at a specific time point, so as to generate a user response probability distribution model; Based on the user response probability distribution model, an advertisement recommendation strategy is obtained by intelligently exploring and making decisions on the optimal advertisement content and the corresponding push timing.

[0008] Optionally, using a Markov decision process to simulate the probability distribution of the user's response at different push times, and calculating the probability distribution of each advertisement content being accepted by the user at a specific time point, so as to generate a user response probability distribution model, includes: Using a Markov decision process, the state space of the obtained user response probability is processed to generate a state space set describing the user state, wherein the state space set includes the user's interest preference, the scenario mode in which the user is located, and the historical interaction record; According to the state space set, the action space of the user's response probability is processed to form an action space set, wherein the action space includes advertisement content and push timing; Based on the state space set and the action space set, the response probability distribution of the user at different push timings is simulated to obtain response value evaluation results of different advertisement contents and push timings; According to the plurality of response value evaluation results, the optimal response value evaluation result is solved by dynamic programming, and the advertisement content and push time corresponding to the optimal response value evaluation result are used as the optimal advertisement push solution; Based on the optimal advertisement push solution, the probability distribution of each advertisement content being accepted by the user at a specific time point is calculated and optimized to generate a user response probability distribution model.

[0009] Optionally, the optimal push timing corresponding to the optimal advertising content is calculated from the optimal advertising content and the corresponding push timing, and the optimal advertising content is pushed to the user based on the optimal push timing; and a real-time feedback mechanism is established based on Bayesian optimization, and the user's feedback information is used to optimize the advertising recommendation strategy using an online learning framework to obtain an adjusted advertising recommendation strategy, including: Using deep reinforcement learning, we simulate the probability distribution of user responses to historical advertising content and the corresponding push timing to obtain the optimal push timing for advertising. Based on the optimal push timing, the optimal advertising content is personalized pushed to generate an advertising recommendation scheme; Based on the advertising recommendation scheme, the Markov decision process is used to intelligently explore the optimal advertising content and the push timing to generate a personalized advertising recommendation strategy; According to the personalized advertising recommendation strategy, using Bayesian optimization, a real-time feedback mechanism is established to collect, analyze and process the feedback information of the user's pushed advertisements to obtain an optimized advertising recommendation strategy; Based on the optimized advertising recommendation strategy, the latest feedback information of the user is absorbed and updated by using an online learning framework to obtain an adjusted advertising recommendation strategy.

[0010] Optionally, a multi-dimensional effect evaluation system integrating hierarchical reinforcement learning and multi-objective evolutionary algorithm is applied to comprehensively evaluate the advertising effect of the adjusted advertising recommendation strategy according to traditional marketing goals and user experience indicators to generate advertising effect evaluation results, including: According to the obtained user experience indicators, feature extraction and enhancement processing are performed on the user experience indicators to obtain a set of evaluation indicators for advertising effects; Based on the evaluation indicator set, a decision framework constructed by hierarchical reinforcement learning is used, a deep network is used in high-level decisions, and a long short-term memory network is used in low-level decisions to capture the user's immediate behavior pattern, and obtain a user interest prediction based on the high-level decisions and the low-level decisions; Based on the user interest prediction, the decision-making process of the advertisement push is optimized using the multi-objective evolutionary algorithm to generate a multi-dimensional effect evaluation system integrating the hierarchical reinforcement learning and the multi-objective evolutionary algorithm; Based on the multi-dimensional effect evaluation system, verify the effectiveness of the adjusted advertising recommendation strategy to generate a verification result; Based on the verification result, according to the traditional marketing goal and the user experience indicator, the advertising effect of the adjusted advertising recommendation strategy is comprehensively evaluated to generate an advertising effect evaluation result.

[0011] Optionally, based on the multi-dimensional effect evaluation system, the effectiveness of the obtained adjusted advertising recommendation strategy is verified to generate a verification result, including: Using Gaussian process regression or Bayesian optimization, the adjusted advertising recommendation strategy is evaluated according to the framework and indicators provided by the multi-dimensional effect evaluation system to obtain a preliminary performance evaluation table; Based on the preliminary performance evaluation table, verify the adjusted advertisement recommendation strategy, and obtain preliminary verification results based on the user behavior by dynamically allocating traffic to the user; Based on the preliminary verification results, the actual impact of the adjusted advertising recommendation strategy is evaluated using a causal relationship framework to obtain an evaluation result based on the causal relationship; Based on the evaluation results, a real-time feedback form of the user is generated to ensure that the updated user feedback information is timely incorporated into the user feedback form; Based on the user's feedback form, the adjusted advertisement recommendation strategy is adjusted using an online learning framework to generate a verification result.

[0012] Optionally, the method of modeling the user's interest preferences based on the scenario mode and the behavior characteristics by using a graph neural network combined with a node importance scoring mechanism and an attention mechanism to generate a personalized interest dynamic graph includes: Utilizing the real-time interactive signal of the user, analyzing and processing the received browsing behavior, clicking behavior and dwell time of the user to obtain user behavior data with time series characteristics; Analyzing the real-time interaction signal according to the user behavior data to obtain the scenario mode of the user; Based on the scenario mode and the user's interest preference, a preliminary interest dynamic graph structure is generated; Using a graph neural network and a node importance scoring mechanism, the preliminary interest dynamic graph structure is evaluated, the importance of the nodes to the user's interests is evaluated, and a preliminary interest dynamic graph is generated; The attention mechanism is used to optimize the preliminary interest dynamic graph, dynamically adjust the importance weight of each node, and generate a personalized interest dynamic graph.

[0013] Optionally, based on the personalized interest dynamic graph, a deep reinforcement learning algorithm combining dual Q learning and policy gradient is used, and the response probability distribution of the user at different push timings is simulated through a Markov decision process, and the optimal advertising content and the corresponding push timing are intelligently explored and decided to obtain an advertising recommendation strategy, including: Optionally, based on traditional marketing objectives and user experience indicators, a comprehensive evaluation process is performed on the advertising effect of the adjusted advertising recommendation strategy to generate an advertising effect evaluation result, including: Generate a traditional marketing objective function by combining the user's click-through rate, conversion rate, and return on investment , ensuring the generation of advertising effectiveness evaluation results based on the traditional marketing objectives; In a second aspect, the embodiment of the present application provides an advertising recommendation system based on deep reinforcement learning, including: An identification module receives a user's real-time interaction signal, wherein the interaction signal includes the user's browsing behavior, clicking behavior, and dwell time, analyzes the time series characteristics in the real-time interaction signal to extract the user's behavior characteristics, and identifies the user's scenario mode; A generation module, based on the scenario mode and the behavior characteristics, uses a graph neural network combined with a node importance scoring mechanism and an attention mechanism to model the user's interest preferences to generate a personalized interest dynamic map, wherein the personalized interest dynamic map is used to reflect the user's immediate interest trend, long-term interest trend and potential demand points; An exploration module, based on the personalized interest dynamic graph, uses a deep reinforcement learning algorithm that combines dual Q learning and policy gradient, and simulates the response probability distribution of the user at different push timings through a Markov decision process, and intelligently explores and makes decisions on the best advertising content and the corresponding push timing to obtain an advertising recommendation strategy, wherein the advertising recommendation strategy includes the best advertising content and the corresponding push timing; The adjustment module calculates the optimal push timing corresponding to the optimal advertising content from the optimal advertising content and the corresponding push timing, and pushes the optimal advertising content to the user based on the optimal push timing; and establishes a real-time feedback mechanism based on Bayesian optimization, and uses the user's feedback information to optimize the advertising recommendation strategy using an online learning framework to obtain an adjusted advertising recommendation strategy; The evaluation module applies a multi-dimensional effect evaluation system that integrates hierarchical reinforcement learning and multi-objective evolutionary algorithms, conducts a comprehensive evaluation of the advertising effect of the adjusted advertising recommendation strategy based on traditional marketing goals and user experience indicators, and generates an advertising effect evaluation result.

[0014] In a third aspect, an embodiment of the present application provides a computing device, comprising a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement an advertising recommendation method based on deep reinforcement learning as described in the first aspect above.

[0015] In a fourth aspect, an embodiment of the present application provides a computer storage medium storing a computer program. When the computer program is executed by a computer, it implements an advertising recommendation method based on deep reinforcement learning as described in the first aspect.

[0016] In an embodiment of the present application, a real-time interaction signal of a user is received, the real-time interaction signal including the browsing behavior, click behavior and dwell time of the user, the time series characteristics in the real-time interaction signal are analyzed to extract the behavioral characteristics of the user, and the scenario mode in which the user is located is identified; according to the scenario mode and the behavioral characteristics, a graph neural network is used in combination with a node importance scoring mechanism and an attention mechanism to model the interest preferences of the user to generate a personalized interest dynamic map, the personalized interest dynamic map is used to reflect the user's immediate interest trends, long-term interest trends and potential demand points; based on the personalized interest dynamic map, a deep reinforcement learning algorithm combining dual Q learning and policy gradient is used, and a Markov decision process is used to simulate the response probability of the user at different push timings distribution, intelligently explore and make decisions on the best advertising content and the corresponding push timing to obtain an advertising recommendation strategy, which includes the best advertising content and the corresponding push timing; from the best advertising content and the corresponding push timing, calculate the optimal push timing corresponding to the optimal advertising content, and push the optimal advertising content to the user based on the optimal push timing; and establish a real-time feedback mechanism based on Bayesian optimization, and use the online learning framework to use the user's feedback information to optimize the advertising recommendation strategy to obtain an adjusted advertising recommendation strategy; apply a multi-dimensional effect evaluation system that integrates hierarchical reinforcement learning and multi-objective evolutionary algorithms, and comprehensively evaluate the advertising effect of the adjusted advertising recommendation strategy according to traditional marketing goals and user experience indicators to generate an advertising effect evaluation result.

[0017] The technical solution of this application has the following beneficial effects: Through real-time interactive signal processing and situational pattern recognition, the accuracy of understanding user behavior is improved, and it can dynamically adapt to different user scenarios, thereby enhancing the response speed and accuracy of the recommendation system. Through graph neural networks combined with node importance scoring mechanisms and attention mechanisms, user interest preferences are modeled to generate personalized interest dynamic maps, ensuring the creation of richer and more dynamic user interest models, improving the accuracy and relevance of personalized recommendations, and enhancing user experience. Through intelligent exploration and decision processing, it ensures that the optimal advertising recommendation strategy is found at different times and scenarios to maximize long-term user value. Through continuous optimization and real-time feedback, the effectiveness and efficiency of advertising recommendations are improved, forming a closed-loop optimization process and improving the flexibility and responsiveness of the overall system. A multi-dimensional effect evaluation system is established, providing a scientific and systematic evaluation framework to ensure that the optimization direction of the recommendation strategy is in line with business goals while taking into account user experience, achieving balanced and sustainable development.

[0018] Furthermore, through personalized interest dynamic graphs, the system can more comprehensively understand users' immediate interest trends, long-term interest trends, and potential demand points, thereby providing advertising content that is more in line with user needs; analyze the user's situational mode to ensure that the recommended content not only meets the user's interest preferences, but also adapts to their current situation, thereby improving the relevance of the recommendation and user experience; use the Markov decision process to simulate the probability distribution of user responses under different push timings to ensure that the advertisements are not only appropriate in content, but also pushed at the most appropriate time, thereby maximizing the user's response rate and engagement; combine dual Q learning with the deep reinforcement learning algorithm of policy gradient to ensure efficient exploration and decision-making in complex environments, find the optimal combination of advertising content and push timing, and improve the intelligence level of the system.

[0019] Furthermore, through the Markov decision process, the system can more accurately predict the user's response probability at different push times, improving the effectiveness of the recommendation strategy. This method takes into account the user's historical behavior and current scenario, enhancing the reliability of the prediction; by constructing a state space set and an action space set, it ensures that the system can describe the user's state and possible behavior choices in detail, providing a basis for more accurate simulation and evaluation; dynamic programming solves the optimal response value evaluation result, ensuring that each recommendation decision is based on the principle of optimization, maximizing the comprehensive effect of advertising content and push timing; through the optimal advertising push solution, the system can adjust and optimize the probability distribution of each advertising content being accepted by the user at a specific time point in real time, forming a closed-loop optimization process, ensuring that the system can quickly adapt to market changes and changes in user behavior, and maintain the timeliness and effectiveness of the recommendation strategy; the generation of the user response probability distribution model provides a scientific and systematic framework for the evaluation of advertising effectiveness, ensuring that the recommendation strategy not only meets business goals but also pays attention to user experience.

[0020] These and other aspects of the present application will become more clearly understood in the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 A flowchart of an advertisement recommendation method based on deep reinforcement learning provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of an advertisement recommendation system based on deep reinforcement learning provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0024] In some of the processes described in the specification and claims of this application and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0026] Figure 1 A flowchart of an advertisement recommendation method based on deep reinforcement learning is provided for an embodiment of the present application, such as Figure 1 As shown, the method includes: 101. Receive a real-time interaction signal from a user, wherein the real-time interaction signal includes browsing behavior, clicking behavior, and dwell time of the user, analyze time series characteristics in the real-time interaction signal to extract behavioral features of the user, and identify a scenario mode of the user; In this step, receiving real-time interaction signals from users means that the advertising recommendation system collects user behavior data through various channels, including user browsing behavior, click behavior, and dwell time. These real-time interaction signals are the key to understanding users' current interests and needs. By analyzing these time series characteristics, we can extract user behavior characteristics and identify the user's situational mode, thereby more accurately capturing the user's immediate needs.

[0027] In an application embodiment, in an e-commerce customer service platform, when a customer enters the help center and starts browsing FAQs, the system immediately starts recording the customer's browsing path and stay time. If a customer frequently clicks on the link about the return policy and stays on the page for a long time, the system will recognize that the customer may be considering a return or encountering return-related problems. At this time, the system can not only provide customers with detailed return guidelines, but also actively suggest that customers contact online customer service for further help. This rapid response mechanism based on real-time interactive signals significantly improves customer satisfaction and service efficiency.

[0028] 102. Based on the scenario mode and the behavior characteristics, a graph neural network combined with a node importance scoring mechanism and an attention mechanism is used to model the user's interest preferences to generate a personalized interest dynamic map, where the personalized interest dynamic map is used to reflect the user's immediate interest trend, long-term interest trend, and potential demand points; In this step, the generation of a personalized interest dynamic graph is the modeling result of the user's interest preferences, which not only reflects the user's immediate and long-term interest trends, but also reveals the user's potential needs. This graph combines a graph neural network to capture the complex relationship between user interests, and introduces a node importance scoring mechanism and an attention mechanism to ensure that the model can focus on the most critical interest nodes while adapting to changes in user interests over time.

[0029] In the application embodiment, the customer service system constructs an interest dynamic graph containing nodes such as "return policy", "order status", and "product warranty" based on the customer's browsing path and residence time in the help center. The system found that the "return policy" node has the highest importance score, indicating that customers have a strong interest in this topic. At the same time, the attention mechanism shows that customers are currently most concerned about the "return process". Based on this graph, the system not only provides detailed return process instructions, but also actively prompts customers to check whether there are any products that meet the return conditions, and guides customers to complete the return application, which greatly simplifies the customer's operation steps and improves the service experience.

[0030] 103. Based on the personalized interest dynamic graph, a deep reinforcement learning algorithm combining dual Q learning and policy gradient is used, and the response probability distribution of the user at different push timings is simulated through a Markov decision process, and the optimal advertising content and the corresponding push timing are intelligently explored and decided to obtain an advertising recommendation strategy, wherein the advertising recommendation strategy includes the optimal advertising content and the corresponding push timing; In this step, intelligent exploration of the best advertising content and push timing refers to the use of deep reinforcement learning algorithms combined with Markov decision processes to simulate the probability distribution of user responses at different push timings to find the optimal combination of advertising content and push timing. This method not only considers the relevance and attractiveness of the advertising content, but also optimizes the choice of push timing to ensure that the advertisement can reach the user at the most appropriate time and maximize the user response rate.

[0031] In the application embodiment, the system simulates the effects of various customer service solutions and their push timings based on the customer's personalized interest dynamic map. For example, while the customer is viewing the return policy, the system evaluates the response probabilities of various solutions such as immediately pushing a detailed return guide and later reminding the customer to check the return progress. Through the Markov decision process simulation, the system found that the response probability of immediately pushing a detailed return guide was the highest, so this solution was selected. In actual application, when the customer visits the return policy page again, the system immediately pops up a detailed return guide and provides a return application entrance, allowing customers to quickly solve the problem, improving service efficiency and customer satisfaction.

[0032] 104. Calculate the optimal push timing corresponding to the optimal advertising content from the optimal advertising content and the corresponding push timing, and push the optimal advertising content to the user based on the optimal push timing; establish a real-time feedback mechanism based on Bayesian optimization, and use the user's feedback information to optimize the advertising recommendation strategy using an online learning framework to obtain an adjusted advertising recommendation strategy; In this step, the optimal push timing and push of the optimal advertising content are calculated based on the user response probability distribution model, and the optimal push plan is finally determined and executed through intelligent exploration and decision-making of the best advertising content and the corresponding push timing. This process ensures that the advertising content is not only highly matched with the user's interests, but also pushed at the most appropriate time, thereby maximizing the advertising effect and user engagement.

[0033] In the application embodiment, the e-commerce customer service system calculates the best time to push the detailed return guide immediately when the customer visits the return policy page for the first time based on the customer's interest dynamic map and response probability distribution model for the return policy. Therefore, when the customer clicks on the return policy link for the first time, the system immediately pops up a detailed return guide and provides a return application entry. After the customer successfully completes the return application, the system also collects the customer's satisfaction evaluation through a real-time feedback mechanism, so as to further optimize the push strategy in similar situations in the future. This series of intelligent push measures not only solves the customer's problems, but also enhances the customer's trust and loyalty.

[0034] 105. Apply a multi-dimensional effect evaluation system integrating hierarchical reinforcement learning and multi-objective evolutionary algorithm to comprehensively evaluate the advertising effect of the adjusted advertising recommendation strategy according to traditional marketing goals and user experience indicators, and generate advertising effect evaluation results; In this step, the comprehensive evaluation process is based on a multi-dimensional effect evaluation system that integrates hierarchical reinforcement learning and multi-objective evolutionary algorithms. According to traditional marketing goals and user experience indicators (such as satisfaction, dwell time, and engagement scores), the adjusted advertising recommendation strategy is comprehensively evaluated. This process generates advertising effect evaluation results to ensure that the recommendation strategy not only meets business goals but also attaches importance to user experience, achieving a harmonious unity between the two.

[0035] In the application example, the e-commerce customer service system conducted a comprehensive evaluation of the detailed return guide that was previously pushed based on the customer's real-time feedback and historical data. The evaluation results show that customers are highly satisfied with the immediate push of detailed return guides, the stay time after the problem is solved is significantly reduced, and the proportion of repeat customers has increased. Based on these evaluation results, the system decides to continue to adopt this push strategy in similar situations in the future, and further optimize the push content and timing, in order to continuously improve customer service efficiency and customer satisfaction. This series of evaluation and optimization measures ensures the continuous improvement of customer service quality and brings a better service experience to customers.

[0036] In summary, through the above steps 101-106, the method not only improves the accuracy and timeliness of advertising recommendations, but also enhances the intelligence level of the system and user experience. It provides a scientific and systematic solution for the field of advertising recommendation, significantly improves advertising effects and customer satisfaction, and achieves a win-win situation for commercial benefits and user experience. Throughout the process, from data collection to intelligent recommendation to effect evaluation, each link is closely connected to form a complete intelligent recommendation closed loop, ensuring the continuous optimization and improvement of the system.

[0037] Optionally, in step 103, based on the personalized interest dynamic graph, a deep reinforcement learning algorithm combining dual Q learning and policy gradient is used, and the response probability distribution of the user at different push timings is simulated through a Markov decision process, and the optimal advertising content and the corresponding push timing are intelligently explored and decided to obtain an advertising recommendation strategy, including: The personalized interest dynamic graph is used to analyze and process the user's interest preferences and the user's situation mode to obtain input data; based on the input data, a deep reinforcement learning algorithm combining dual Q learning and policy gradient is used to simulate and process multiple advertising content and corresponding push timing combinations to obtain potential impact evaluation results; based on the potential impact evaluation results, a Markov decision process is used to simulate the user's response probability distribution under different push timings, and the probability distribution of each advertising content being accepted by the user at a specific time point is calculated to generate a user response probability distribution model; based on the user response probability distribution model, an advertising recommendation strategy is obtained by intelligently exploring and making decisions on the optimal advertising content and the corresponding push timing.

[0038] Optionally, using a Markov decision process to simulate the probability distribution of the user's response at different push times, and calculating the probability distribution of each advertisement content being accepted by the user at a specific time point, so as to generate a user response probability distribution model, includes: The state space of the obtained user's response probability is processed by using a Markov decision process to generate a state space set describing the user's state, wherein the state space set includes the user's interest preference, the scenario mode and the historical interaction record; based on the state space set, the action space of the user's response probability is processed to form an action space set, wherein the action space includes the advertisement content and the push timing; based on the state space set and the action space set, the response probability distribution of the user at different push timings is simulated to obtain the response value evaluation results of different advertisement contents and push timings; based on multiple response value evaluation results, the optimal response value evaluation result is solved by using dynamic programming, and the advertisement content and push time corresponding to the optimal response value evaluation result are used as the optimal advertisement push scheme; based on the optimal advertisement push scheme, the probability distribution of each advertisement content being accepted by the user at a specific time point is calculated and optimized to generate a user response probability distribution model.

[0039] In this step, a personalized interest dynamic graph is generated based on the user's behavioral characteristics and scenario patterns to ensure that the model can focus on the most critical interest nodes. The simulation process is based on the input data. The system uses a deep reinforcement learning algorithm that combines dual Q learning and policy gradient to simulate a variety of advertising content and corresponding push timing combinations. This process aims to evaluate the potential impact of different solutions in different scenarios. Based on the response value evaluation results of different advertising content and push timing obtained from the simulation, the system uses dynamic programming to solve the optimal response value evaluation result.

[0040] In the application embodiment, a personalized interest dynamic map is used to analyze the user's interest preferences and scenario patterns in detail, and input data that can be used for subsequent simulation processing is extracted. Based on the input data, a deep reinforcement learning algorithm combining dual Q learning and policy gradient is used to simulate a variety of advertising content and push timing combinations to ensure that potential impact assessment results are obtained. The advertising recommendation strategy is obtained by intelligently exploring and making decisions on the best advertising content and push timing. At the same time, the optimal response value assessment result is solved through dynamic planning, the optimal advertising push plan is determined, and the probability distribution of each advertising content being accepted by the user at a specific time point is further optimized.

[0041] In an e-commerce customer service platform, when a customer enters the help center to browse FAQs, the system begins to monitor the customer's interactive behavior in real time, collecting the customer's browsing path, the specific question links clicked, and the dwell time on each page. The system first identifies the customer's current demand scenario (such as considering a return), and then generates a map reflecting the customer's immediate and long-term interests based on the personalized interest dynamic map. The system uses the data in the map, combined with the deep reinforcement learning algorithm of double Q learning and policy gradient, to simulate the effects of multiple customer service solutions at different push times. By simulating the probability distribution of customer responses at different push times, the system obtains the response value evaluation results of different solutions. The system establishes a real-time feedback mechanism based on Bayesian optimization, and adjusts the push strategy in time to ensure that each push can achieve the best effect. Through this series of intelligent and personalized recommendations and services, the system not only solves customers' problems, but also enhances customers' trust and loyalty, achieving a win-win situation in terms of commercial benefits and user experience.

[0042] Optionally, in step 104, the optimal push timing corresponding to the optimal advertising content is calculated from the optimal advertising content and the corresponding push timing, and the optimal advertising content is pushed to the user based on the optimal push timing; and a real-time feedback mechanism is established based on Bayesian optimization, and the user's feedback information is used to optimize the advertising recommendation strategy using an online learning framework to obtain an adjusted advertising recommendation strategy, including: Using deep reinforcement learning, the user response probability distribution of historical advertising content and the corresponding push timing is simulated to obtain the optimal push timing of the advertisement; based on the optimal push timing, the optimal advertising content is personalized and pushed to generate an advertising recommendation plan; based on the advertising recommendation plan, the Markov decision process is used to intelligently explore the optimal advertising content and the push timing to generate a personalized advertising recommendation strategy; according to the personalized advertising recommendation strategy, Bayesian optimization is used to establish a real-time feedback mechanism to collect and analyze the feedback information of the user's pushed advertisements to obtain an optimized advertising recommendation strategy; based on the optimized advertising recommendation strategy, the online learning framework is used to absorb and update the latest feedback information of the user to obtain an adjusted advertising recommendation strategy.

[0043] In this step, the historical data is simulated by simulating the probability distribution of user responses and using a deep reinforcement learning algorithm to predict the probability distribution of user responses under different advertising content and push timings, ensuring that the system finds the optimal push timing. Based on the results of the simulation, the optimal push time for each advertising content is determined. By real-time monitoring of user feedback information such as clicks, dwell time, and satisfaction, the system can understand the effectiveness of the advertisement in a timely manner and adjust the recommendation strategy accordingly. Using the online learning framework, the system can quickly absorb and update the latest user feedback information, continuously optimize the advertising recommendation strategy, and ensure that the system has adaptive capabilities and can quickly respond to market changes and changes in user needs.

[0044] In the application embodiment, deep reinforcement learning is used to simulate the probability distribution of user responses to historical advertising content and the corresponding push timing to ensure that the system finds the optimal push timing for advertising. Based on the calculated optimal push timing, personalized push processing is performed on the optimal advertising content to ensure that the system generates an advertising recommendation plan. Based on the generated personalized advertising recommendation strategy, a real-time feedback mechanism is established using Bayesian optimization to collect and analyze user feedback information on pushed advertising to obtain an optimized advertising recommendation strategy. Based on the optimized advertising recommendation strategy, an online learning framework is used to absorb and update the user's latest feedback information to obtain an adjusted advertising recommendation strategy, forming a closed-loop optimization process.

[0045] In an e-commerce customer service platform, when a customer enters the help center to browse FAQs, the system begins to monitor the customer's interactive behavior in real time. The system first identifies the customer's current demand scenario (such as considering a return), and then generates a map reflecting the customer's immediate and long-term interests based on the personalized interest dynamic map. The system uses deep reinforcement learning to simulate the probability distribution of user responses to historical advertising content and the corresponding push timing. Based on the optimized advertising recommendation strategy, the system uses an online learning framework to absorb and update the latest feedback information from customers. For example, if a customer raises new questions or needs in the future return process, the system will quickly adjust the push strategy to provide more accurate service suggestions.

[0046] Optionally, in step 105, a multi-dimensional effect evaluation system integrating hierarchical reinforcement learning and multi-objective evolutionary algorithm is applied to comprehensively evaluate the advertising effect of the adjusted advertising recommendation strategy according to traditional marketing goals and user experience indicators to generate advertising effect evaluation results, including: According to the obtained user experience indicators, feature extraction and enhancement processing are performed on the user experience indicators to obtain a set of evaluation indicators for advertising effects; based on the set of evaluation indicators, a decision framework constructed by hierarchical reinforcement learning is used, deep networks are used in high-level decisions, and long short-term memory networks are used in low-level decisions to capture the user's immediate behavior pattern and obtain user interest predictions based on the high-level decisions and the low-level decisions; based on the user interest prediction, the decision-making process of the advertising push is optimized using the multi-objective evolutionary algorithm to generate a multi-dimensional effect evaluation system that integrates the hierarchical reinforcement learning and the multi-objective evolutionary algorithm; based on the multi-dimensional effect evaluation system, the effectiveness of the obtained adjusted advertising recommendation strategy is verified to generate a verification result; based on the verification result, the advertising effect of the adjusted advertising recommendation strategy is comprehensively evaluated according to the traditional marketing goals and the user experience indicators to generate an advertising effect evaluation result.

[0047] Optionally, based on the multi-dimensional effect evaluation system, the effectiveness of the obtained adjusted advertising recommendation strategy is verified to generate a verification result, including: Using Gaussian process regression or Bayesian optimization, the adjusted advertising recommendation strategy is evaluated according to the framework and indicators provided by the multi-dimensional effect evaluation system to obtain a preliminary performance evaluation table; based on the preliminary performance evaluation table, the adjusted advertising recommendation strategy is verified, and through the dynamic traffic allocation of the user, a preliminary verification result based on the user behavior is obtained; based on the preliminary verification result, the actual impact of the adjusted advertising recommendation strategy is evaluated using a causal relationship framework to obtain an evaluation result based on the causal relationship; based on the evaluation result, a real-time feedback form for the user is generated to ensure that the updated user feedback information is promptly included in the user's feedback form; based on the user's feedback form, the adjusted advertising recommendation strategy is adjusted using an online learning framework to generate a verification result.

[0048] In this step, user experience indicators include user satisfaction, dwell time, engagement scores, etc., which reflect users' immediate reactions and long-term feedback to advertisements or customer service. User experience indicators include user satisfaction, dwell time, engagement scores, etc., which reflect users' immediate reactions and long-term feedback to advertisements or customer service. Gaussian process regression or Bayesian optimization is used to conduct a preliminary evaluation of the adjusted advertising recommendation strategy based on the framework and indicators provided by the multi-dimensional effect evaluation system. The online learning framework continuously absorbs user feedback and updates the recommendation strategy to form a closed-loop optimization process. The causal relationship framework is used to evaluate the real impact of the adjusted advertising recommendation strategy to ensure the scientificity and accuracy of the evaluation results.

[0049] In the application embodiment, the user experience indicators are processed through data analysis techniques (such as principal component analysis, feature selection, etc.), the most representative and predictive features are extracted, and enhanced to ensure the accuracy and reliability of the evaluation. The obtained feature set is used to construct a set of evaluation indicators for advertising effectiveness. Combining the results of high-level and low-level decisions, the system can more comprehensively predict changes in user interests and ensure that the recommended content is in line with the user's long-term preferences and adapts to their current needs. Through steps such as preliminary performance evaluation tables, dynamic traffic allocation verification, and causal relationship evaluation, the verification results are finally generated to ensure the effectiveness of the adjusted advertising recommendation strategy.

[0050] In an e-commerce customer service platform, when a customer enters the help center to browse FAQs, the system begins to monitor the customer's interactive behavior in real time, collecting the customer's browsing path, the specific question links clicked, and the time spent on each page. The system extracts and enhances the user experience indicators to extract the most representative features, such as the customer's attention to specific issues and the efficiency of solving them. The advertising effect of the adjusted advertising recommendation strategy is comprehensively evaluated to generate advertising effect evaluation results. For example, the system found that immediately pushing detailed return guides not only improved customer satisfaction, but also increased the platform's conversion rate and return on investment. Based on these evaluation results, the system decided to continue to adopt this push strategy in similar situations in the future, and further optimize the push content and timing, in order to continuously improve customer service efficiency and customer satisfaction.

[0051] Optionally, in step 102, based on the scenario mode and the behavior characteristics, a graph neural network combined with a node importance scoring mechanism and an attention mechanism is used to model the user's interest preferences to generate a personalized interest dynamic graph, including: The user's real-time interaction signal is used to analyze and process the received browsing behavior, clicking behavior and dwell time of the user to obtain user behavior data with time series characteristics; based on the user behavior data, the real-time interaction signal is parsed to obtain the scenario mode of the user; based on the scenario mode, a preliminary interest dynamic graph structure is generated based on the user's interest preference; using a graph neural network and a node importance scoring mechanism, the preliminary interest dynamic graph structure is evaluated and processed to evaluate the importance of the nodes to the user's interest and generate a preliminary interest dynamic graph; using an attention mechanism, the preliminary interest dynamic graph is optimized to dynamically adjust the importance weights of each node to generate a personalized interest dynamic graph.

[0052] In this step, real-time interaction signals include users’ browsing behavior (pages visited and content), click behavior (specific links or buttons clicked), and dwell time (length of stay on the page). Time series characteristics: refers to the change pattern of these behavioral data over time, such as the user’s browsing path, click frequency, and the time interval between each visit. The preliminary interest dynamic graph structure is based on the user’s interest preferences to build a preliminary graph structure that reflects the user’s interest relationship. The nodes in the graph structure represent different points of interest, and the edges represent the correlation between points of interest. The node importance scoring mechanism assigns a weight to each node, indicating the importance of the node to the user’s interest. Through the evaluation process, the system can identify which points of interest are most critical to the user.

[0053] In the application embodiment, by analyzing and processing real-time interactive signals, the browsing behavior, click behavior and dwell time of the received users are analyzed and processed to ensure that user behavior data with time series characteristics are obtained. Through evaluation processing and importance scoring, the importance of each node in the interest graph structure to the user's interest is evaluated. The graph neural network can capture the complex relationship between nodes, and combined with the node importance scoring mechanism, ensure that the model focuses on the most critical nodes of interest. Through optimization processing and personalized adjustment, the importance weights of each node are dynamically adjusted. The attention mechanism enables the system to focus on the nodes that the user is currently most interested in, ensuring that the interest graph reflects the user's immediate interest trends.

[0054] In an e-commerce customer service platform, when a customer enters the help center to browse FAQs, the system analyzes and processes the customer's real-time interactive signals. Using graph neural networks, the system combines the node importance scoring mechanism to evaluate the importance of each node in the interest graph structure to the customer's interest. For example, the system found that the "return policy" node had the highest importance score, indicating that the customer has a strong interest in this topic. At the same time, the system also identifies other related points of interest, such as "order status" and "product warranty." The system uses the attention mechanism to optimize the preliminary interest dynamic graph and dynamically adjust the importance weight of each node. Based on the optimized personalized interest dynamic graph, the system not only provides a detailed description of the return process, but also actively prompts customers to check whether there are products that meet the return conditions, and guides customers to complete the return application, which greatly simplifies the customer's operation steps and improves the service experience.

[0055] In the advertising recommendation system, selecting the best advertising content and push timing is a complex and dynamic process. Traditional recommendation algorithms often have difficulty capturing users' immediate interest changes and scenario patterns, resulting in poor recommendation results. In order to improve the accuracy and timeliness of recommendations, a deep reinforcement learning algorithm combining dual Q learning and policy gradient was introduced. This method simulates the probability distribution of user responses at different push timings, intelligently explores the best advertising content and its corresponding push timing, and thus optimizes the advertising recommendation strategy.

[0056] Optionally, based on the personalized interest dynamic graph, a deep reinforcement learning algorithm combining dual Q learning and policy gradient is used, and the response probability distribution of the user at different push timings is simulated through a Markov decision process, and the optimal advertising content and the corresponding push timing are intelligently explored and decided to obtain an advertising recommendation strategy, including: Based on the personalized interest dynamic graph, dual Q learning is used to use two independent Q value estimators in each interaction. and , to reduce the maximization bias; The calculation formula of double Q learning is: ; in, Represents two independent Value estimator, is the adaptive learning rate, is a discount factor reflecting the importance of future rewards, is a time decay factor that adjusts the importance of future rewards over time, and are the current state and action respectively. and are the next state and action respectively, It is The Q-value estimator is in the state and the actions The state-action value under is in the state and the actions The instant reward obtained under the above conditions reflects the user's response intensity to the specific advertising content and push timing. It means that in the double Q learning, the two Q value estimators are in the next state The actions described in The minimum value under Based on the double Q learning, the preliminary advertising recommendation strategy is obtained by optimizing the deep reinforcement learning algorithm using the policy gradient , to obtain the probability distribution of the action selected in a given state; Strategy gradient calculation formula: ; in, is the policy gradient, which represents the policy The objective function The policy gradient From the time step arrive The instant reward The sum of About The expected value of According to the strategy The generated state and action pairs, Indicates that in the state Next select action The probability of is the probability In the state Select the action below The logarithmic gradient of the probability distribution of is the initial advertising recommendation strategy The state-action value combination under is the time step, is the baseline function, , About the action Expected value; Combining the calculation results of the dual Q learning and the strategy gradient, intelligent exploration is performed on the best advertising content and the corresponding push timing to obtain an advertising recommendation strategy. ; Intelligent exploration calculation formula: ; in, is the optimal advertising strategy function, which refers to the state Select the action below The probability distribution of Refers to finding the preliminary advertising recommendation strategy that maximizes the internal expression , It's about the trajectory The expected value of According to the preliminary advertising recommendation strategy The state pairs and the action pairs generated, From the time step arrive The sum of is the discount factor Power, is the weight of the click behavior, is the weight of the dwell time, Indicates that in the state Take the action below After the user clicks the ad, is the log-transformed residence time; According to the optimal advertising recommendation strategy, the system is guided to select the best advertising content and push it to the user at the corresponding push time, thereby ensuring the effectiveness and accuracy of the advertising recommendation.

[0057] Double Q learning reduces maximization bias by using two independent Q-value estimators, making the evaluation results more stable. The policy gradient method can optimize the initial ad recommendation strategy obtained by adjusting the policy parameters to maximize long-term rewards. The final intelligent exploration formula comprehensively considers multiple indicators such as click behavior and dwell time to ensure that the recommendation strategy not only focuses on short-term click-through rate, but also attaches importance to user engagement and satisfaction.

[0058] The following is a brief explanation of the design reasons for each item in the formula: The reason for designing the calculation formula of double Q learning: Double Q learning uses two independent Q value estimators to reduce the maximization bias, making the evaluation results more stable, ensuring that the recommendation system can accurately capture the user's immediate interest changes, provide personalized service recommendations, and ultimately achieve a win-win situation in commercial benefits and user experience.

[0059] Formula expression: Indicates in status and actions Double Q learning uses two independent Q value estimators to reduce the maximization bias, making the evaluation results more stable; It represents instant rewards and quantifies the user's behavioral response. It reflects the user's response intensity to specific advertising content and push timing, and is an important part of the optimization goal. Indicates that the minimum of the two Q-value estimators is selected to reduce the overestimation bias and ensure a more conservative and stable evaluation result; The main purpose is to combine short-term and long-term rewards while reducing estimation bias and ensuring the stability and reliability of evaluation results; By combining immediate feedback with long-term planning, the estimation bias is reduced and stability is improved. That is, the actual immediate reward plus the expected future reward (the discounted minimum Q value estimate) minus the estimated value of the action taken in the current state, this difference reflects the difference between the model's prediction of future rewards and the actual situation. Effective updates of state-action values ​​are achieved.

[0060] The design reason of the policy gradient calculation formula: The policy gradient calculation formula is designed to adjust the policy parameters , so that the objective function Maximize, thereby maximizing the long-term cumulative reward. This helps the recommendation system to meet the user's immediate needs while also taking into account the user's long-term interests and satisfaction, ensuring that the system updates the policy parameters more effectively to increase the probability of selecting high-reward actions.

[0061] Formula expression: Refers to the policy gradient, which means about the policy parameters The objective function The gradient of , which is used to guide parameter updates to maximize long-term rewards; Represents the time step arrive The sum of the immediate rewards, as part of the objective function, aims to maximize the cumulative reward; It's about the trajectory The expected value of takes into account all possible interaction paths, ensuring that strategy optimization is based on comprehensive data distribution; Indicates in status Next select action The probability of reflects the behavioral tendency under the current strategy; represents the logarithmic gradient of the probability distribution and is used to adjust the policy parameters to make high-reward actions more likely to be chosen; Refers to the initial advertising recommendation strategy The state-action value combination under the condition measures the expected benefit after taking a certain action. , is the baseline function used to reduce variance, About the action to improve the stability of the gradient estimate.

[0062] The design reason of the intelligent exploration calculation formula: It represents the initial advertising recommendation strategy to find the maximum internal expression , which is the optimal advertising recommendation strategy. This process ensures that the system can find the one that can best improve user engagement and satisfaction among all possible strategies, ensuring the final generation of the advertising recommendation strategy.

[0063] Formula expression: By introducing click behavior, the system can quickly respond to the user's immediate interest changes and ensure that the recommended content meets the user's current needs. Allows flexible adjustment of business needs. Click-through rate is an important indicator and is given a higher weight to ensure that the system prioritizes content with high click-through rate. The dwell time part reflects the user's interest depth and engagement in the recommended content. A longer dwell time usually means that the user has a higher interest or satisfaction in the content. refers to the optimal advertising strategy function, refers to the state Next select action The probability distribution of Refers to finding the initial advertising recommendation strategy that maximizes the internal expression ; It's about the trajectory The expected value of Based on the initial advertising recommendation strategy Generated state and action pairs; is the discount factor The second act emphasizes the importance of recent rewards; Is the state of the user clicking on the ad; represents the log-transformed residence time, which is used to smooth the effect of long residence times.

[0064] The following is a brief explanation of how to obtain the parameters of the formula: Represents two independent Value estimator, It is an adaptive learning rate, which sets the initial value based on experimental data or experience and dynamically adjusts it during actual operation; is a discount factor, determined through experiments and historical data analysis, and usually set to a value close to but less than 1 (such as 0.9) to emphasize the importance of recent rewards. is a time decay factor that adjusts the importance of the future reward over time; and are the current state and action respectively. and are the next said state and said action respectively; Sequences of state-action pairs generated through simulated or real user interactions.

[0065] In the embodiment of the present application, in an e-commerce customer service platform, it is assumed that we have the following specific values: Discount Factor Click behavior weight Dwell time weight Adaptive learning rate (initial value) Time decay factor Instant Rewards : Calculated based on user clicks and dwell time Implementation steps: initialization: Set discount factor , click behavior weight , residence time weight , adaptive learning rate , time decay factor .

[0066] Collect user interaction data: Record when customers browse the return policy page, click on specific issue links, and how long they stay on each page.

[0067] Build a personalized interest dynamic graph: Based on the customer's real-time interaction signals, analyze the customer's current situation mode (such as considering returning a product) and generate a personalized interest dynamic map.

[0068] Applying Double Q Learning: Using two independent Q-value estimators and , update the Q value in each interaction: ; Assume that after an interaction, the immediate reward (The user clicks on the detailed return guide), the Q value is updated.

[0069] Policy Gradient Optimization: Optimize initial advertising recommendation strategy , using the policy gradient formula to adjust the policy parameters to maximize the long-term reward.

[0070] Intelligent exploration of optimal strategies: Combining the results of double Q learning and policy gradient, intelligent exploration is performed: ; Assume that after a certain exploration, the system finds that the click rate of users who are immediately pushed detailed return instructions is , the average residence time is 120 seconds, then we can calculate: ; After calculation, the system concluded that immediately pushing detailed return guidelines is the optimal strategy, because the user click-through rate and dwell time under this strategy are both high, indicating that users have a strong demand for this information, while also improving user satisfaction and service efficiency.

[0071] In the ad recommendation system, evaluating the ad effect requires not only considering traditional marketing goals (such as click-through rate, conversion rate, and return on investment), but also user experience indicators (such as user satisfaction, dwell time, and engagement score). In order to comprehensively evaluate the ad effect of the adjusted ad recommendation strategy, the traditional marketing objective function is combined and user experience metrics , and generate the final advertising effect evaluation results .

[0072] Optionally, based on traditional marketing objectives and user experience indicators, a comprehensive evaluation process is performed on the advertising effect of the adjusted advertising recommendation strategy to generate an advertising effect evaluation result, including: Generate a traditional marketing objective function by combining the user's click-through rate, conversion rate, and return on investment , ensuring the generation of advertising effectiveness evaluation results based on the traditional marketing objectives; The calculation formula of traditional marketing objective function is: ; in, is the traditional marketing objective function, is the click-through rate, which represents the ratio of the number of times the ad is displayed to the number of times it is clicked. is the conversion rate, which indicates the proportion of conversions actually completed after clicking on the ad. is the return on investment, which represents the ratio of the revenue and cost brought by the advertisement. are weights of the click-through rate, the conversion rate and the return on investment, respectively. The weights are adjusted according to specific business needs to reflect the importance of different indicators. Used to avoid excessive reward inflation when the click rate is high. Indicates the importance of conversion rate; Generate the user experience index based on the user satisfaction, dwell time and user engagement score , ensuring the generation of comprehensive evaluation results based on the user experience indicators; User experience index calculation formula: ; in, is user satisfaction, obtained through the user survey and rating method, is the dwell time, which represents the average dwell time of the user on the advertisement or related page. is the user engagement score, ensuring that engagement behavior is given appropriate weight, are the weights of the user satisfaction, the dwell time and the engagement score, respectively. The weights can be adjusted according to specific business needs to reflect the importance of different indicators. Used to emphasize the importance of user satisfaction, logarithmic function Used to avoid excessive rewards due to long stays; According to the traditional marketing objectives and the user experience indicators, a comprehensive evaluation process is performed on the advertising effect of the adjusted advertising recommendation strategy to generate an advertising effect evaluation result; Advertising effectiveness evaluation results The calculation formula is: ; in, is the evaluation value of the traditional marketing goal, including the click-through rate, the conversion rate and the return on investment, is the evaluation value of the user experience index, including the user satisfaction, the dwell time and the engagement score, is the optimal advertising strategy to obtain.

[0073] By introducing indicators with different weights, we ensure that the system can balance short-term marketing goals with long-term user experience. We use logarithmic and exponential functions to process certain indicators to prevent evaluation bias caused by extreme values. By using exponential functions for key indicators to highlight their importance, we ensure that the system prioritizes these factors and generates the final advertising effect evaluation results.

[0074] The following is a brief explanation of the design reasons for each item in the formula: The design reason of the traditional marketing objective function: by balancing short-term and long-term effects, avoiding the impact of extreme values ​​and emphasizing key indicators, it ensures that the system can efficiently respond to users' immediate needs and quickly adjust recommendation strategies.

[0075] Formula expression: The click-through rate is an important indicator for measuring the attractiveness of an ad, but a high click-through rate does not always mean an improvement in ad effectiveness. It can effectively prevent excessive inflation of rewards when the click rate is high, maintain the stability of the evaluation, and avoid evaluation deviations caused by a very small number of extreme values, ensuring that the evaluation results are more reasonable and stable; Represents the conversion rate part, which directly reflects the actual effect of the advertisement, that is, the proportion of users who actually complete the conversion after clicking the advertisement. Use exponential function The importance of conversion rate is emphasized to ensure that the system prioritizes advertising content that can actually lead to actual conversions.

[0076] The reason for designing the user experience index calculation formula is to ensure that the system not only pays attention to the user's immediate response, but also pays attention to the user's long-term engagement, and provides a more comprehensive service experience by improving user satisfaction, reasonably evaluating the dwell time and paying attention to user engagement.

[0077] Formula expression: Refers to the user satisfaction part, which is an important indicator to measure the user's acceptance of advertising content. It uses an exponential function Emphasizes the importance of user satisfaction, ensuring that the system prioritizes advertising content that satisfies users. The index will significantly amplify the impact of high satisfaction, making ads with high satisfaction receive more attention, which helps improve the overall user experience and loyalty. Represents the residence time portion, using a logarithmic function It can smooth the impact of long-term stay, avoid deviations caused by very few extreme values, and ensure that the evaluation results are more reasonable and stable, so that the user's participation depth can be more reasonably evaluated; The user engagement scoring part takes into account a variety of user interaction behaviors, which helps to improve the overall user experience and loyalty, and the system can evaluate user engagement more comprehensively.

[0078] The reason for designing the calculation formula for advertising effect evaluation results: By comprehensively evaluating advertising effects and guiding the optimal advertising strategy, we ensure that the recommendation system can make scientific and reasonable decisions based on accurate data analysis, continuously optimize the recommendation strategy, and achieve the best advertising effect and user experience.

[0079] Formula expression: is the evaluation value of the traditional marketing goal, including the click-through rate, the conversion rate and the return on investment, which is calculated by the above formula Calculate and obtain; represents the evaluation value of the user experience index, including the user satisfaction, the dwell time and the engagement score, which is given by the above formula Calculate and obtain; It is the optimal advertising strategy obtained through optimization of reinforcement learning algorithm to ensure the effectiveness and accuracy of the recommendation strategy.

[0080] The following is a brief explanation of how to obtain the parameters of the formula: Click-through rate and conversion rate : Calculated from ad impressions and click data, usually obtained through real-time monitoring and historical data analysis; Return on Investment : Calculated by the ratio of revenue to costs from advertising, usually based on financial data and sales records; Customer satisfaction : Obtained through user surveys, ratings or net promoter scores; Dwell time : Extracted from user behavior data, including browsing behavior, dwell time, and page interaction; User Engagement Rating : Quantified through user interaction behavior, usually obtained through interaction data provided by analysis platforms; Weight Parameters : According to specific business needs, through Testing and user research to determine the optimal weight configuration.

[0081] In the embodiment of the present application, in an e-commerce customer service platform, it is assumed that we have the following specific values: Click-through rate Conversion rate Return on Investment Customer satisfaction (Out of 5) Dwell time Second User Engagement Rating :85 (out of 100) The weights are set as follows: Calculation steps: Traditional marketing objective function : ; User experience metrics : ; Advertising effectiveness evaluation results : ; Through the above calculations, the system concluded that the current advertising strategy performed well in terms of traditional marketing goals and user experience indicators, especially the high scores of user satisfaction and engagement, indicating that users have a good acceptance of advertising content and willingness to participate. This series of intelligent recommendation measures has brought customers a better service experience and enhanced their trust and loyalty.

[0082] Figure 2 A structural diagram of an advertisement recommendation system based on deep reinforcement learning is provided for an embodiment of the present application. Figure 2 As shown, the device comprises: The identification module 21 receives a real-time interaction signal of a user, wherein the interaction signal includes the browsing behavior, click behavior and dwell time of the user, analyzes the time series characteristics in the real-time interaction signal to extract the behavior characteristics of the user, and identifies the scenario mode of the user; A generation module 22, based on the scenario mode and the behavior characteristics, uses a graph neural network combined with a node importance scoring mechanism and an attention mechanism to model the user's interest preferences to generate a personalized interest dynamic map, wherein the personalized interest dynamic map is used to reflect the user's immediate interest trend, long-term interest trend and potential demand points; The exploration module 23 uses a deep reinforcement learning algorithm that combines dual Q learning and policy gradient based on the personalized interest dynamic graph, and simulates the response probability distribution of the user at different push timings through a Markov decision process, and intelligently explores and makes decisions on the best advertising content and the corresponding push timing to obtain an advertising recommendation strategy, wherein the advertising recommendation strategy includes the best advertising content and the corresponding push timing; The adjustment module 24 calculates the optimal push timing corresponding to the optimal advertising content from the optimal advertising content and the corresponding push timing, and pushes the optimal advertising content to the user based on the optimal push timing; and establishes a real-time feedback mechanism based on Bayesian optimization, and uses the user's feedback information to optimize the advertising recommendation strategy using an online learning framework to obtain an adjusted advertising recommendation strategy; Evaluation module 25 applies a multi-dimensional effect evaluation system integrating hierarchical reinforcement learning and multi-objective evolutionary algorithm, performs a comprehensive evaluation on the advertising effect of the adjusted advertising recommendation strategy according to traditional marketing goals and user experience indicators, and generates an advertising effect evaluation result.

[0083] Figure 2 The advertising recommendation system based on deep reinforcement learning can be executed Figure 1 The implementation principle and technical effect of the advertising recommendation method based on deep reinforcement learning described in the illustrated embodiment will not be described in detail. The specific manner in which each module and unit performs operations in the advertising recommendation system based on deep reinforcement learning in the above embodiment has been described in detail in the embodiment of the method, and will not be elaborated here.

[0084] In one possible design, Figure 2 An advertisement recommendation system based on deep reinforcement learning in the illustrated embodiment can be implemented as a computing device, such as Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32; The storage component 31 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 32 .

[0085] The processing component 32 is used to: receive a real-time interaction signal from a user, the real-time interaction signal including the browsing behavior, click behavior and dwell time of the user, analyze the time series characteristics in the real-time interaction signal to extract the behavior characteristics of the user, and identify the scenario mode of the user; According to the scenario mode and the behavior characteristics, a graph neural network combined with a node importance scoring mechanism and an attention mechanism is used to model the user's interest preferences to generate a personalized interest dynamic map, which is used to reflect the user's immediate interest trends, long-term interest trends, and potential demand points; Based on the personalized interest dynamic graph, a deep reinforcement learning algorithm combining dual Q learning and policy gradient is used, and the response probability distribution of the user at different push timings is simulated through a Markov decision process, and the optimal advertising content and the corresponding push timing are intelligently explored and decided to obtain an advertising recommendation strategy, which includes the optimal advertising content and the corresponding push timing; From the optimal advertising content and the corresponding push timing, the optimal push timing corresponding to the optimal advertising content is calculated, and the optimal advertising content is pushed to the user based on the optimal push timing; and a real-time feedback mechanism is established based on Bayesian optimization, and the user's feedback information is used to optimize the advertising recommendation strategy using an online learning framework to obtain an adjusted advertising recommendation strategy; A multi-dimensional effect evaluation system integrating hierarchical reinforcement learning and multi-objective evolutionary algorithm is applied to comprehensively evaluate the advertising effect of the adjusted advertising recommendation strategy according to traditional marketing goals and user experience indicators to generate advertising effect evaluation results.

[0086] The processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component may also be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0087] The storage component 31 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0088] Of course, the computing device may also include other components, such as input / output interfaces, display components, communication components, etc.

[0089] The input / output interface provides an interface between the processing component and the peripheral interface module, which may be an output device, an input device, etc.

[0090] The communication component is configured to facilitate, among other things, wired or wireless communications between the computing device and other devices.

[0091] Among them, the computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. In this case, the computing device can refer to a cloud server, and the above-mentioned processing components, storage components, etc. can be basic server resources rented or purchased from the cloud computing platform.

[0092] The present application also provides a computer storage medium storing a computer program, wherein the computer program can achieve the above-mentioned Figure 1 An advertisement recommendation method based on deep reinforcement learning in the illustrated embodiment.

[0093] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0094] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0095] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An advertisement recommendation method based on deep reinforcement learning, characterized in that: include: receiving a real-time interaction signal of a user, wherein the real-time interaction signal includes the browsing behavior, clicking behavior and dwell time of the user, analyzing the time series characteristics in the real-time interaction signal to extract the behavior characteristics of the user, and identifying the scenario mode of the user; According to the scenario mode and the behavior characteristics, a graph neural network combined with a node importance scoring mechanism and an attention mechanism is used to model the user's interest preferences to generate a personalized interest dynamic map, which is used to reflect the user's immediate interest trends, long-term interest trends, and potential demand points; Based on the personalized interest dynamic graph, a deep reinforcement learning algorithm combining dual Q learning and policy gradient is used, and the response probability distribution of the user at different push timings is simulated through a Markov decision process, and the optimal advertising content and the corresponding push timing are intelligently explored and decided to obtain an advertising recommendation strategy, which includes the optimal advertising content and the corresponding push timing; Calculating the optimal push timing corresponding to the optimal advertising content from the optimal advertising content and the corresponding push timing, and pushing the optimal advertising content to the user based on the optimal push timing; A real-time feedback mechanism is established based on Bayesian optimization, and the user feedback information is used to optimize the advertising recommendation strategy using an online learning framework to obtain an adjusted advertising recommendation strategy; A multi-dimensional effect evaluation system integrating hierarchical reinforcement learning and multi-objective evolutionary algorithm is applied to comprehensively evaluate the advertising effect of the adjusted advertising recommendation strategy according to traditional marketing goals and user experience indicators to generate advertising effect evaluation results.

2. The method according to claim 1, characterized in that Based on the personalized interest dynamic graph, a deep reinforcement learning algorithm combining dual Q learning and policy gradient is used, and the response probability distribution of the user at different push timings is simulated through a Markov decision process, and the optimal advertising content and the corresponding push timing are intelligently explored and decided to obtain an advertising recommendation strategy, including: Using the personalized interest dynamic graph, the user's interest preference and the user's scenario mode are analyzed and processed to obtain input data; based on the input data, a deep reinforcement learning algorithm combining dual Q learning and policy gradient is used to simulate and process a variety of advertising content and corresponding push timing combinations to obtain potential impact assessment results; According to the potential impact assessment result, using the Markov decision process to simulate the user's response probability distribution at different push timings, calculating the probability distribution of each type of the advertising content being accepted by the user at a specific time point, so as to generate a user response probability distribution model; Based on the user response probability distribution model, an advertisement recommendation strategy is obtained by intelligently exploring and making decisions on the optimal advertisement content and the corresponding push timing.

3. The method according to claim 2, characterized in that The Markov decision process is used to simulate the probability distribution of the user's response at different push times, and the probability distribution of each advertisement content being accepted by the user at a specific time point is calculated to generate a user response probability distribution model, including: Using a Markov decision process, the state space of the obtained user response probability is processed to generate a state space set describing the user state, wherein the state space set includes: the user's interest preference, the scenario mode in which he is in, and the historical interaction record; According to the state space set, the action space of the user's response probability is processed to form an action space set, wherein the action space includes advertisement content and push timing; Based on the state space set and the action space set, the response probability distribution of the user at different push timings is simulated to obtain response value evaluation results of different advertisement contents and push timings; According to the plurality of response value evaluation results, the optimal response value evaluation result is solved by dynamic programming, and the advertisement content and push time corresponding to the optimal response value evaluation result are used as the optimal advertisement push solution; Based on the optimal advertisement push solution, the probability distribution of each advertisement content being accepted by the user at a specific time point is calculated and optimized to generate a user response probability distribution model.

4. The method according to claim 1, characterized in that: The optimal advertising content and the corresponding push timing are used to calculate the optimal push timing corresponding to the optimal advertising content, and the optimal advertising content is pushed to the user based on the optimal push timing; A real-time feedback mechanism is established based on Bayesian optimization, and the user feedback information is used to optimize the advertising recommendation strategy using an online learning framework to obtain an adjusted advertising recommendation strategy, including: Using deep reinforcement learning, we simulate the probability distribution of user responses to historical advertising content and the corresponding push timing to obtain the optimal push timing for advertising. Based on the optimal push timing, the optimal advertising content is personalized pushed to generate an advertising recommendation scheme; Based on the advertising recommendation scheme, the Markov decision process is used to intelligently explore the optimal advertising content and the push timing to generate a personalized advertising recommendation strategy; According to the personalized advertising recommendation strategy, using Bayesian optimization, a real-time feedback mechanism is established to collect, analyze and process feedback information of the user's pushed advertisements to obtain an optimized advertising recommendation strategy; Based on the optimized advertising recommendation strategy, the latest feedback information of the user is absorbed and updated by using an online learning framework to obtain an adjusted advertising recommendation strategy.

5. The method according to claim 1, characterized in that A multi-dimensional effect evaluation system integrating hierarchical reinforcement learning and multi-objective evolutionary algorithm is applied to comprehensively evaluate the advertising effect of the adjusted advertising recommendation strategy according to traditional marketing goals and user experience indicators, and generate advertising effect evaluation results, including: According to the obtained user experience indicators, feature extraction and enhancement processing are performed on the user experience indicators to obtain a set of evaluation indicators for advertising effects; Based on the evaluation indicator set, a decision framework constructed by hierarchical reinforcement learning is used, a deep network is used in high-level decisions, and a long short-term memory network is used in low-level decisions to capture the user's immediate behavior pattern, and obtain a user interest prediction based on the high-level decisions and the low-level decisions; Based on the user interest prediction, the decision-making process of the advertisement push is optimized using the multi-objective evolutionary algorithm to generate a multi-dimensional effect evaluation system integrating the hierarchical reinforcement learning and the multi-objective evolutionary algorithm; Based on the multi-dimensional effect evaluation system, verify the effectiveness of the adjusted advertising recommendation strategy to generate a verification result; Based on the verification result, according to the traditional marketing goal and the user experience indicator, the advertising effect of the adjusted advertising recommendation strategy is comprehensively evaluated to generate an advertising effect evaluation result.

6. The method according to claim 5, characterized in that Based on the multi-dimensional effect evaluation system, the effectiveness of the adjusted advertising recommendation strategy is verified to generate verification results, including: Using Gaussian process regression or Bayesian optimization, the adjusted advertising recommendation strategy is evaluated according to the framework and indicators provided by the multi-dimensional effect evaluation system to obtain a preliminary performance evaluation table; Based on the preliminary performance evaluation table, verify the adjusted advertisement recommendation strategy, and obtain preliminary verification results based on the user behavior by dynamically allocating traffic to the user; Based on the preliminary verification results, the actual impact of the adjusted advertising recommendation strategy is evaluated using a causal relationship framework to obtain an evaluation result based on the causal relationship; Based on the evaluation results, a real-time feedback form of the user is generated to ensure that the updated user feedback information is timely incorporated into the user feedback form; Based on the user's feedback form, the adjusted advertisement recommendation strategy is adjusted using an online learning framework to generate a verification result.

7. The method according to claim 1, characterized in that The method of modeling the user's interest preferences based on the scenario mode and the behavior characteristics by using a graph neural network combined with a node importance scoring mechanism and an attention mechanism to generate a personalized interest dynamic graph includes: Utilizing the real-time interactive signal of the user, analyzing and processing the received browsing behavior, clicking behavior and dwell time of the user to obtain user behavior data with time series characteristics; Analyzing the real-time interaction signal according to the user behavior data to obtain the scenario mode of the user; Based on the scenario mode and the user's interest preference, a preliminary interest dynamic graph structure is generated; Using a graph neural network and a node importance scoring mechanism, the preliminary interest dynamic graph structure is evaluated, the importance of the nodes to the user's interests is evaluated, and a preliminary interest dynamic graph is generated; The attention mechanism is used to optimize the preliminary interest dynamic graph, dynamically adjust the importance weight of each node, and generate a personalized interest dynamic graph.

8. An advertisement recommendation method based on deep reinforcement learning, characterized in that: include: An identification module receives a user's real-time interaction signal, wherein the interaction signal includes the user's browsing behavior, clicking behavior, and dwell time, analyzes the time series characteristics in the real-time interaction signal to extract the user's behavior characteristics, and identifies the user's scenario mode; A generation module, based on the scenario mode and the behavior characteristics, uses a graph neural network combined with a node importance scoring mechanism and an attention mechanism to model the user's interest preferences to generate a personalized interest dynamic map, wherein the personalized interest dynamic map is used to reflect the user's immediate interest trend, long-term interest trend and potential demand points; An exploration module, based on the personalized interest dynamic graph, uses a deep reinforcement learning algorithm that combines dual Q learning and policy gradient, and simulates the response probability distribution of the user at different push timings through a Markov decision process, and intelligently explores and makes decisions on the best advertising content and the corresponding push timing to obtain an advertising recommendation strategy, wherein the advertising recommendation strategy includes the best advertising content and the corresponding push timing; an adjustment module, which calculates an optimal push timing corresponding to the optimal advertising content from the optimal advertising content and the corresponding push timing, and pushes the optimal advertising content to the user based on the optimal push timing; A real-time feedback mechanism is established based on Bayesian optimization, and the user feedback information is used to optimize the advertising recommendation strategy using an online learning framework to obtain an adjusted advertising recommendation strategy; The evaluation module applies a multi-dimensional effect evaluation system that integrates hierarchical reinforcement learning and multi-objective evolutionary algorithms, conducts a comprehensive evaluation of the advertising effect of the adjusted advertising recommendation strategy based on traditional marketing goals and user experience indicators, and generates an advertising effect evaluation result.

9. A computing device, characterized in that It includes a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement an advertising recommendation method based on deep reinforcement learning as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a computer, an advertisement recommendation method based on deep reinforcement learning as described in any one of claims 1 to 7 is implemented.

Images