Book recommendation method and system based on multi-agent reinforcement learning

Abstract

The invention provides a book recommendation method and system based on multi-agent reinforcement learning, and relates to the technical field of book recommendation, and the method comprises the steps: constructing a multi-agent reinforcement learning model, and achieving the information interaction and cooperation between agents through a centralized training architecture; based on the deterministic characteristics of the multi-agent reinforcement learning model, optimistic search is introduced to optimize book search; performing attention weighting optimization on strategy gradient updating through a dominant function to optimize a recommendation strategy; and the strategy parameters of the multiple agents are synchronized through the distributed execution architecture to realize collaborative decision making. According to the method, the multi-agent reinforcement learning models are constructed for mutual assistance and distributed execution, so that the calculation complexity is remarkably reduced, and real-time recommendation of large-scale users and books is supported. On the basis of the deterministic characteristics of the multi-agent reinforcement learning model, optimistic search is introduced to optimize book search, a search algorithm based on quantile optimistic estimation is proposed, and the search efficiency is remarkably improved. And the advantage value searched by the Monte Carlo tree is combined with the strategy gradient, so that the strategy degradation problem of the sparse reward scene is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of book recommendation, and in particular to a book recommendation method and system based on multi-agent reinforcement learning. Background Art

[0002] The Internet has changed people's lifestyles, making their lives and studies more convenient. In recent years, with the widespread popularity of the Internet, the rapid development of e-commerce, and the popularity of digital reading, book recommendation systems have played an important role in helping users discover new books.

[0003] However, existing book recommendation systems have made significant progress in single-agent scenarios, but single-agent models find it difficult to capture group collaboration effects, and exhaustive search cannot generate recommendation strategies in real time. Traditional reinforcement learning algorithms are prone to falling into local optimality due to the lack of effective value guidance. Therefore, they still face challenges in multi-agent collaboration, large-scale action space optimization, and sparse reward scenarios. Summary of the Invention

[0004] The purpose of the present invention is to provide a book recommendation method and system based on multi-agent reinforcement learning to solve the problems mentioned in the above background technology that the existing book recommendation system still faces challenges in multi-agent collaboration, large-scale action space optimization and sparse reward scenarios.

[0005] To achieve the above-mentioned objectives, the present invention provides the following technical solutions: a book recommendation method based on multi-agent reinforcement learning, comprising the following steps: constructing a multi-agent reinforcement learning model, and realizing information interaction and collaboration among agents through a centralized training architecture; introducing optimistic search to optimize book search based on the deterministic characteristics of the multi-agent reinforcement learning model; performing attention weighting on the policy gradient update through the advantage function to optimize the recommendation strategy; and synchronizing the policy parameters of the multi-agents through a distributed execution architecture to realize multi-agent collaborative decision-making.

[0006] Optionally, the multi-agent includes: a user agent, which is used to receive user characteristics and environmental characteristics as input, and output the distribution probability of personalized recommended books; a book agent, which is used to receive book characteristics and user interest distribution, and output the probability of whether to actively recommend the book; and an environmental agent, which is used to manage global resource allocation; wherein, the user characteristics include user ID embedding, user historical behavior sequence and user interest distribution; the book characteristics include book ID embedding, book text content, book category tags, and real-time popularity of books; and the environmental characteristics include a global popular book list, user interest trends, and system resource limitations.

[0007] Optionally, the steps of realizing information interaction and collaboration among intelligent agents through a centralized training architecture specifically include: predicting the global state transition and reward after multi-agent interaction through a centralized dynamic model, wherein the centralized dynamic model includes a local dynamic model with shared parameters and a multi-head attention mechanism.

[0008] Optionally, the centralized dynamic model includes: a representation function h θ : Used to map each agent's local observations into hidden states, supporting distributed execution; communication function e θ : Used to aggregate global collaborative information through the multi-head attention mechanism to generate additional features of the agent; dynamic function g θ : Used to combine the current state, action and communication features to predict the hidden state and immediate reward at the next moment; prediction function V θ , P θ : Used to output global value estimates and individual policy distributions.

[0009] Optionally, the step of introducing optimistic search to optimize book search based on the deterministic characteristics of the multi-agent reinforcement learning model specifically includes: based on the deterministic environment model in the multi-agent reinforcement learning model, introducing an optimistic bias term in the Monte Carlo tree search process to optimize book search.

[0010] Optionally, the step of introducing an optimistic bias term in the Monte Carlo tree search process to optimize the book search specifically includes: introducing an optimistic bias term in the selection phase of the Monte Carlo tree search, whose calculation formula is: OS(λ)=Q(s,a)+λ·Model_Value(s,a); wherein OS(λ) is the optimistic search priority, Q(s,a) is the action value, Model_Value(s,a) is the long-term value of the action predicted by the model, and λ is a hyperparameter that controls the degree of optimism and is used to balance exploration and utilization; in the deterministic environment model, the average value calculation of the Monte Carlo tree search is replaced by optimistic value estimation, and its calculation formula is: Where, is an optimistic estimate, is the optimistic return quantile at depth d, λ is the depth discount factor, and the exploration and exploitation are balanced by adjusting λ and ρ.

[0011] Optionally, the step of introducing an optimistic bias term in the Monte Carlo tree search process to optimize the book search also includes: when expanding nodes, using the centralized dynamic model to predict future state transfers and rewards to directly evaluate the potential value of the action; in the simulation stage, if the model has high certainty, using the trajectory predicted by the centralized dynamic model instead of random simulation; judging the value of the path predicted by the centralized dynamic model according to a preset threshold, if it is a low-value path, pruning the low-value path, and if it is a high-value path, depth-first expanding the high-value path.

[0012] Optionally, the step of weighting the policy gradient update by the advantage function to optimize the recommendation strategy specifically includes: using the advantage function as a weight to weight the loss function; in the backtracking phase of the Monte Carlo tree search, recording the simulated reward output by the dynamic function in the centralized dynamic model of each action and the predicted value output by the prediction function; using the simulated reward and the predicted value to calculate the advantage function as the weight for the policy gradient update; when the reward is sparse, supplementing the real reward signal with the virtual reward predicted by the centralized dynamic model.

[0013] Optionally, the centralized dynamic model is trained using a multi-agent consistency loss function to minimize prediction error.

[0014] On the other hand, the present invention also provides a book recommendation system based on multi-agent reinforcement learning, including: a construction module for constructing a multi-agent reinforcement learning model, and realizing information interaction and collaboration between agents through a centralized training architecture; a search module for introducing optimistic search to optimize book search based on the deterministic characteristics of the multi-agent reinforcement learning model; a recommendation module for weighting the policy gradient update through the advantage function to optimize the recommendation strategy; and a decision module for synchronizing the policy parameters of the multi-agent through a distributed execution architecture to realize multi-agent collaborative decision-making.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] This application significantly reduces computational complexity by building a multi-agent reinforcement learning model with mutual assistance and distributed execution, enabling real-time recommendations for large-scale users and books. Based on the deterministic nature of this multi-agent reinforcement learning model, it introduces optimistic search to optimize book search and proposes a search algorithm based on quantile optimistic estimation, significantly improving search efficiency. For the first time, it combines the advantage value of Monte Carlo tree search with policy gradients to address the problem of policy degradation in sparse reward scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Schematic diagram of the process steps of the present invention.

[0018] Figure 2 Schematic diagram of the system structure of the present invention.

[0019] In the figure: 10-construction module, 20-search module, 30-recommendation module, 40-decision module. DETAILED DESCRIPTION

[0020] The following will provide a clear and complete description of the solutions of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0021] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0022] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of this application refers to the presence of features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0023] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0024] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not imply the order of execution. The order of execution of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of this application.

[0025] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0026] Please refer to Figure 1 The present invention provides a book recommendation method based on multi-agent reinforcement learning, which includes the following steps: constructing a multi-agent reinforcement learning model, and realizing information interaction and collaboration among agents through a centralized training architecture; introducing optimistic search to optimize book search based on the deterministic characteristics of the multi-agent reinforcement learning model; performing attention weighting on the policy gradient update through the advantage function to optimize the recommendation strategy; synchronizing the policy parameters of the multi-agent through a distributed execution architecture to realize multi-agent collaborative decision-making.

[0027] Specifically, this application significantly reduces computational complexity by constructing a multi-agent reinforcement learning model with mutual assistance and distributed execution, enabling real-time recommendations for large-scale users and books. Based on the deterministic nature of this multi-agent reinforcement learning model, an optimistic search optimization for book searches is introduced, and a search algorithm based on quantile optimistic estimation is proposed, significantly improving search efficiency. For the first time, the advantage value of Monte Carlo tree search is combined with policy gradients to address the problem of policy degradation in sparse reward scenarios.

[0028] In some embodiments, the multi-agent includes: a user agent, which receives user characteristics and environmental characteristics as input and outputs the distribution probability of personalized book recommendations; a book agent, which receives book characteristics and user interest distribution and outputs the probability of whether to actively recommend the book; an environmental agent, which manages global resource allocation; wherein, the user characteristics include user ID embedding, user historical behavior sequence and user interest distribution; the book characteristics include book ID embedding, book text content, book category tags, and real-time popularity of books; the environmental characteristics include a global popular book list, user interest trends, and system resource limitations.

[0029] Specifically, the user agent is responsible for personalized recommendations. Each user or user group acts as an agent, receiving user characteristics and environmental characteristics as input, and outputting the distribution probability of personalized recommended books. The book agent treats each category of books (such as science fiction, literature, or popular books) as an independent agent, provides content characteristics, receives book characteristics and user interest distribution, and outputs the probability of actively recommending the book. The environmental agent is responsible for managing the global state (such as popular trends, user interest distribution, etc.) and reward distribution. This application solves the inherent limitations of a single agent in personalization, content quality, and real-time performance through the division of labor and collaboration among multiple agents, ultimately realizing an efficient, accurate, and scalable book recommendation system.

[0030] In some embodiments, the step of realizing information interaction and collaboration between intelligent agents through a centralized training architecture specifically includes: predicting the global state transition and reward after multi-agent interaction through a centralized dynamic model, and the centralized dynamic model includes a local dynamic model with shared parameters and a multi-head attention mechanism.

[0031] Specifically, the centralized dynamic model is shared by all agents, predicting the global state transition and reward after multi-agent interaction, and inputting the joint state S of all agents = {s u , s b , s e}, output the next moment global state S′ and reward R; where s u is the state of the user agent, s b is the state of the book agent, s e is the state of the environment agent, and the reward R includes indicators such as user click rate and system diversity.

[0032] In some embodiments, the centralized dynamic model includes: a representation function h θ : Used to map each agent's local observations into hidden states, supporting distributed execution; communication function e θ : Used to aggregate global collaborative information through the multi-head attention mechanism to generate additional features of the agent; dynamic function g θ : Used to combine the current state, action and communication features to predict the hidden state and immediate reward at the next moment; prediction function V θ , P θ : Used to output global value estimates and individual policy distributions.

[0033] Specifically, the representation function, the communication function, the dynamic function, and the prediction function share the same parameter θ, which can promote collaborative optimization among multiple agents, enable multiple agents to adapt to each other during the training process, improve overall consistency, and adapt to the approximate independence of multiple agents.

[0034] In some embodiments, the step of introducing optimistic search to optimize book search based on the deterministic characteristics of the multi-agent reinforcement learning model specifically includes: based on the deterministic environment model in the multi-agent reinforcement learning model, introducing an optimistic bias term in the Monte Carlo tree search process to optimize book search.

[0035] In some embodiments, the step of introducing an optimistic bias term in the Monte Carlo tree search process to optimize book search specifically includes: introducing an optimistic bias term in the selection phase of the Monte Carlo tree search, whose calculation formula is: OS(λ)=Q(s,a)+λ·Model_Value(s,a); where OS(λ) is the optimistic search priority, Q(s,a) is the action value, Model_Value(s,a) is the long-term value of the action predicted by the model, and λ is a hyperparameter that controls the degree of optimism and is used to balance exploration and utilization; in a deterministic environment model, the average value calculation of the Monte Carlo tree search is replaced by optimistic value estimation, and its calculation formula is: Where, is an optimistic estimate, is the optimistic return quantile at depth d, λ is the depth discount factor, and the exploration and exploitation are balanced by adjusting λ and ρ.

[0036] Specifically, the optimistic search is used to optimize the multi-agent book search. In the selection phase, the user agent uses the improved UCB to select books, and the calculation formula is: UCB u (a) = Q u (s u , a)+λ u Model_Value(s u , a), where UCB u (a) The upper confidence bound for the user agent to select action a, Q u (s u , a) is the user agent in state s u The expected cumulative reward value Q of selecting action a is u is the optimistic hyperparameter of the user agent, which is used to control the balance between exploration and utilization. u ,a) is the value prediction of action a by the centralized dynamic model, such as long-term return.

[0037] Specifically, book agents compete for recommendation positions and evaluate their own value through the attention mechanism. The calculation formula is: UCB b (a) = Q b (s b , a)+λ b Popularity(s b ), where UCBb (a) The upper confidence bound for the book agent to choose action a, Q b (s b , a) is the book agent in state s b The expected cumulative reward value Q of selecting action a is b is the optimistic hyperparameter of the book agent, which is usually smaller than the optimistic hyperparameter of the user agent. Popularity(s b ) are book popularity indicators, such as real-time click-through rate, popularity ranking, etc.

[0038] In some embodiments, the step of introducing an optimistic bias term in the Monte Carlo tree search process to optimize the book search also includes: when expanding nodes, using the centralized dynamic model to predict the transfer and reward of future states to directly evaluate the potential value of the action; in the simulation stage, if the model certainty is high, using the trajectory predicted by the centralized dynamic model instead of random simulation; judging the value of the path predicted by the centralized dynamic model according to a preset threshold, if it is a low-value path, pruning the low-value path, and if it is a high-value path, depth-first expanding the high-value path.

[0039] Specifically, the centralized dynamic model predicts the global state S′ and reward R after the interaction of multiple agents. If the popularity of the book predicted by the centralized dynamic model matches the user's interest well, the path is expanded first. If the popularity of the book predicted by the centralized dynamic model matches the user's interest poorly, the path is pruned. During backtracking and updating, the global reward R is distributed to each agent. The calculation formula is: R u =αR+(1-α)R personal , R b =βR+(1-β)R popularity , where R u is the user reward, α is the weight of global and local rewards, R is the global reward, R personal For personal rewards, R b is the book reward, β is the global and local reward weights, R popularity Reward for popularity.

[0040] In some embodiments, the step of weighting the policy gradient update with attention through the advantage function to optimize the recommendation strategy specifically includes: using the advantage function as a weight to weight the loss function; in the backtracking phase of the Monte Carlo tree search, recording the simulated reward output by the dynamic function in the centralized dynamic model of each action and the predicted value output by the prediction function; using the simulated reward and predicted value to calculate the advantage function as the weight for the policy gradient update; when the reward is sparse, supplementing the real reward signal with the virtual reward predicted by the centralized dynamic model.

[0041] Specifically, the advantage function of the user agent is: A u (s u , a u )=Q u (s u , a u )-V u (s u ), where A u For user agent action a u The advantage value, Q u (s u , a u ) is the user agent in state s u Next select action a u The expected cumulative reward value, V u (s u ) is the user agent in state s u The baseline value below.

[0042] Specifically, the advantage function of the book agent is: A b (s b , a b )=Q b (s b , a b )-V b (s b ), where A b Action a for the book agent b The advantage value, Q b (s b ,a b ) is the book agent in state s b Next select action a b The expected cumulative reward value, V b (s b ) is the book agent in state s b The baseline value below.

[0043] Specifically, the global advantage is weighted by attention: Where A global is the global advantage value, which is the weighted sum of the advantages of all agents. i ) is the attention weight, according to the agent state s i Dynamic distribution of importance, A i is the advantage value of the ith agent, such as A for the user or book agent u or A b .

[0044] Specifically, the advantage weighted strategy is used to optimize the decision strategies of the user agent and the book agent. The user agent strategy is optimized as follows: Book strategy optimization is: The global strategy aggregates parameters through federated learning, where is the policy gradient loss function, through the advantage value A u / A b Optimization strategy, π u (a u |s u ) is the user agent in state s u Next select action a u The probability distribution of π b (a b |s b ) is the book agent in state s b Next select action a b The probability distribution of τ u , τ b is the trajectory of the user and book agents, i.e., the search path.

[0045] Specifically, each agent samples data from the search trajectory, calculates the local advantage function, shares the global value information, and uses the Adam optimizer to update the user and book policy networks respectively.

[0046] In some embodiments, the centralized dynamic model is trained using a multi-agent consistency loss function to minimize prediction error.

[0047] Specifically, the calculation formula of the consistency loss function is: Where, is the loss function of the centralized dynamic model to minimize the prediction error, S is the joint state of all agents, A is the joint action set of all agents, S′ is the true joint state at the next moment, R is the true reward at the current moment, ModelS(S, A) is the next state predicted by the dynamic model, Model R (S, A) is the immediate reward predicted by the dynamic model.

[0048] Please refer to Figure 2 On the other hand, the present invention also provides a book recommendation system based on multi-agent reinforcement learning, including: a construction module 10, used to construct a multi-agent reinforcement learning model, and realize information interaction and collaboration between agents through a centralized training architecture; a search module 20, used to introduce optimistic search to optimize book search based on the deterministic characteristics of the multi-agent reinforcement learning model; a recommendation module 30, used to weight the policy gradient update through the advantage function to optimize the recommendation strategy; a decision module 40, used to synchronize the policy parameters of the multi-agent through a distributed execution architecture to realize multi-agent collaborative decision-making.

[0049] Specifically, user agents are sharded by user ID and deployed on edge servers, while book agents are grouped by book category and deployed on central servers. The environment agent manages and coordinates resource allocation globally. The environment agent broadcasts popular book lists and user interest distribution hourly, while the book agent reports popularity changes. User and book agents interact via a query interface, synchronizing model parameters using a parameter server model that periodically aggregates the policy parameters of the user and book agents. A multi-head attention communication mechanism drives the system to only forward high-value information, such as new popular books and sudden changes in user interests. During the centralized training phase, all agents share a centralized dynamic model, updating global parameters through federated learning. User and book agents independently optimize their respective policy networks and exchange information on advantages through a communication module. During the distributed inference phase, user agents independently perform recommendations, relying solely on the environment agent to obtain global trends. Book agents compete for recommendation positions, dynamically adjusting priorities through an attention mechanism.

[0050] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0051] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described method embodiments. Any reference to memory, storage, database, or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0052] The above are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's description and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A book recommendation method based on multi-agent reinforcement learning, characterized in that the steps include: Build a multi-agent reinforcement learning model and achieve information interaction and collaboration between agents through a centralized training architecture; Based on the deterministic nature of the multi-agent reinforcement learning model, optimistic search is introduced to optimize book search; Attention-weighted policy gradient updates are performed through advantage functions to optimize recommendation strategies. The strategic parameters of the multi-agents are synchronized through a distributed execution architecture to achieve multi-agent collaborative decision-making.

2. The book recommendation method based on multi-agent reinforcement learning according to claim 1, characterized in that: The multi-agent comprises: The user agent receives user characteristics and environment characteristics as input and outputs the distribution probability of personalized book recommendations; Book agent, which receives book features and user interest distribution and outputs the probability of actively recommending the book; Environmental agents, which manage global resource allocation; The user features include user ID embedding, user historical behavior sequence and user interest distribution; The book features include book ID embedding, book text content, book category label, and book real-time popularity; The environmental characteristics include global popular book lists, user interest trends, and system resource limitations.

3. The book recommendation method based on multi-agent reinforcement learning according to claim 1, characterized in that: The steps of realizing information interaction and collaboration between agents through a centralized training architecture specifically include: The global state transitions and rewards after multi-agent interactions are predicted through a centralized dynamics model, which includes a local dynamics model with shared parameters and a multi-head attention mechanism.

4. The book recommendation method based on multi-agent reinforcement learning according to claim 3 is characterized in that: The centralized dynamic model includes: Represents the function h θ : Used to map each agent's local observations into hidden states, supporting distributed execution; Communication function e θ : Used to aggregate global collaborative information through a multi-head attention mechanism to generate additional features of the agent; Dynamic function g θ : Used to combine the current state, action and communication features to predict the hidden state and immediate reward at the next moment; Prediction function V θ , P θ : Used to output global value estimates and individual policy distributions.

5. The book recommendation method based on multi-agent reinforcement learning according to claim 4 is characterized in that: The steps of introducing optimistic search to optimize book search based on the deterministic characteristics of the multi-agent reinforcement learning model specifically include: Based on the deterministic environment model in the multi-agent reinforcement learning model, an optimistic bias term is introduced into the Monte Carlo tree search process to optimize book search.

6. The book recommendation method based on multi-agent reinforcement learning according to claim 5, characterized in that: The steps of introducing an optimistic bias term in the Monte Carlo tree search process to optimize the book search specifically include: In the selection phase of Monte Carlo tree search, an optimistic bias term is introduced, and its calculation formula is: OS(λ)=Q(s,a)+λ·Model_Value(s,a); Where OS(λ) is the optimistic search priority, Q(s, a) is the action value, Model_Value(s, a) is the long-term value of the action predicted by the model, and λ is a hyperparameter that controls the degree of optimism and is used to balance exploration and exploitation. In the deterministic environment model, the average value calculation of the Monte Carlo tree search is replaced by optimistic value estimation, and its calculation formula is: Where, is an optimistic estimate, is the optimistic return quantile at depth d, λ is the depth discount factor, and the exploration and exploitation are balanced by adjusting λ and ρ.

7. The book recommendation method based on multi-agent reinforcement learning according to claim 5, characterized in that: The step of introducing an optimistic bias term in the Monte Carlo tree search process to optimize the book search further includes: When expanding nodes, the centralized dynamic model is used to predict future state transitions and rewards, directly evaluating the potential value of actions. In the simulation phase, if the model is highly deterministic, the trajectory predicted by the centralized dynamic model is used instead of the random simulation; The value of the path predicted by the centralized dynamic model is judged according to a preset threshold. If it is a low-value path, the low-value path is pruned; if it is a high-value path, the high-value path is depth-first expanded.

8. The book recommendation method based on multi-agent reinforcement learning according to claim 5, characterized in that: The step of performing attention weighting on the policy gradient update through the advantage function to optimize the recommendation strategy specifically includes: Use the advantage function as the weight and weighted loss function; During the backtracking phase of the Monte Carlo tree search, the simulated rewards output by the dynamic function and the predicted value output by the prediction function in the centralized dynamic model for each action are recorded; Calculate the advantage function using simulated returns and predicted values ​​as the weight for policy gradient updates; When rewards are sparse, virtual rewards predicted by a centralized dynamics model supplement the real reward signal.

9. The book recommendation method based on multi-agent reinforcement learning according to claim 3, characterized in that: The centralized dynamic model is trained using a multi-agent consensus loss function to minimize the prediction error.

10. A book recommendation system based on multi-agent reinforcement learning, characterized in that: include: A building block for constructing multi-agent reinforcement learning models, enabling information exchange and collaboration between agents through a centralized training architecture; A search module for optimizing book searches by introducing optimistic search based on the deterministic nature of the multi-agent reinforcement learning model; The recommendation module is used to weight the policy gradient update through the advantage function to optimize the recommendation strategy; The decision-making module is used to synchronize the strategy parameters of the multi-agent through a distributed execution architecture to achieve multi-agent collaborative decision-making.

Images