A q-network control method and device, and a readable storage medium

Abstract

The application discloses a Q network control method and device and a readable storage medium, and relates to the electronic information field. In the Q network control method provided by the application, a preset behavior is a behavior randomly selected in a behavior space A or an optimal behavior selected in the behavior space A by using a database operator. A plurality of experience samples are extracted from an experience pool, and corresponding expected benefits are calculated based on the database operator. Finally, the parameters and vector indexes of the Q network and the parameters and vector indexes of a target Q network are updated according to the expected benefits. The database operator is an operator realized based on an approximate nearest neighbor search technology. Compared with the previous training control method, the approximate nearest neighbor search technology is introduced in the scheme to obtain the database operator, so that the search range is greatly reduced. Compared with the previous DQN which traverses all the action spaces, the training and decision efficiency is increased, and the scheme can be applied to large-scale discrete behavior scenes.

Description

Technical Field

[0001] This application relates to the field of electronic information, and in particular to a Q network control method, apparatus and readable storage medium. Background Technology

[0002] In recent years, with the development of computer and information technology, the computing power and intelligent functions of computers have become increasingly powerful. One of the core elements of intelligence is reinforcement learning, which consists of two parts: an agent and the environment. In reinforcement learning, the agent's goal is to maximize the gains from the environment. Q-learning is a classic reinforcement learning algorithm, using a table to represent Q(x, a). However, this method is not suitable for environments with high-dimensional state (action) spaces or continuous state (action) spaces. Therefore, Deep Q-Networks (DQN) employ deep neural networks to approximate... Where θ represents the parameters of the Q network, DQN enables the agent to learn a better policy and has stronger generalization ability compared to using a simple functional form to approximate Q(s,a).

[0003] Let S be the number of interactions between the agent and the environment during training, and M be the size of the action space. After each interaction, the agent draws B samples from the experience pool. Then, after each interaction, DQN traverses the entire action space at least B times, and the training complexity of DQN is O(SBM). During decision-making, DQN employs a greedy strategy; given a state S, DQN needs to traverse the entire action space to find the optimal action under state S. Therefore, the decision complexity of DQN is O(M). Consequently, DQN is difficult to apply in large-scale discrete behavior scenarios. For example, in recommendation scenarios, the state is the user, and the action space consists of all items, typically reaching tens or even hundreds of millions of actions. If training efficiency is low, the model may fail to converge for a long time, significantly impacting business metrics. Furthermore, low decision efficiency makes it difficult to meet the real-time requirements of recommendation systems.

[0004] Given the above-mentioned technologies, finding a Q-network control method that can be applied to large-scale discrete behavior scenarios is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a Q-network control method, apparatus, and readable storage medium to address the problem that DQN is difficult to apply in large-scale discrete behavior scenarios. For example, in recommendation scenarios, the action space consists of all items, and the action space usually reaches tens of millions or even hundreds of millions. If the training efficiency is low, the model will not converge for a long time, which will have a significant impact on business metrics. If the decision-making efficiency is low, it will be difficult to meet the real-time requirements of the recommendation system.

[0006] To address the above problems, this application provides a Q network control method, comprising:

[0007] The parameters of the Q-network and the target Q-network are initialized, and the vector indices of the Q-network and the target Q-network are established.

[0008] Obtain the current environment state. If the current environment state is empty or terminated, reset the running environment and obtain the initial state of the running environment as the updated environment state.

[0009] Select and execute a preset behavior to obtain environmental feedback and the updated environmental state; wherein, the preset behavior is a behavior randomly selected in behavior space A or the optimal behavior selected in behavior space A using database operators;

[0010] The preset behavior, the environmental feedback, the current environmental state, and the updated environmental state are stored as experience samples in the experience pool; wherein, the experience pool is a storage unit for storing each of the experience samples.

[0011] Multiple experience samples are extracted from the experience pool, and the corresponding expected returns are calculated based on the database operator; wherein, the database operator is an operator implemented based on the approximate nearest neighbor search technique;

[0012] Update the parameters and vector indexes of the Q network and the parameters and vector indexes of the target Q network according to the expected returns.

[0013] Preferably, the Q-network and the target Q-network employ adaptive behavioral representations.

[0014] Preferably, the initialization of the parameters of the Q-network and the target Q-network, and the establishment of the vector indices of the Q-network and the target Q-network, includes:

[0015] Randomly initialize the parameters θ of the Q network = (θ s θ a W1, ..., W K ), where θ s θ is a parameter in the state representation module. a These are the parameters in the Softmax layer, W1, ..., W K The behavior representation module contains K embedding matrices, and the target Q-network and the Q-network have the same network structure, with parameters initialized to...

[0016] The vector index IndexQ of the Q network and the vector index IndexTargetQ of the target Q network are established using the following steps;

[0017] For any action a, connect a on W1, ..., W K The corresponding basis vectors are obtained

[0018]

[0019] Connect a in The corresponding basis vectors are obtained

[0020]

[0021] The vector indices IndexQ and IndexTargetQ are established as follows:

[0022]

[0023]

[0024] Where M is the size of the behavior space, BuildIndex takes all key-value pairs as input and outputs a vector index that stores the inputs, and BuildIndex has a time complexity of O(M log M).

[0025] Preferably, the step of extracting multiple experience samples from the experience pool and calculating the corresponding expected returns based on the database operator includes:

[0026] Based on empirical samples (s) i a i r i ,s′ i ) Calculate state-behavior(s) i a i Expected return y under ) i ; where s i For the current state, a i For the preset behavior, r i For the environmental feedback, s′ i The updated environmental state;

[0027] If s′ i If it is a terminated state, then y i =r i Otherwise: In the target Q-network, calculate the state representation. Weights of basis vectors Construct the query vector:

[0028]

[0029] The query results are as follows:

[0030]

[0031] Wherein, Query is the query operator, and the input of the query operator is the index structure and the query vector. If the size of the index structure is M, then the query operator finds the target in O(log M) time. Make: get Afterwards, the expected return y i The calculation is as follows:

[0032]

[0033] Preferably, updating the parameters and vector indices of the Q-network according to the expected return includes:

[0034] The gradient dθ of the parameter θ of the Q network is calculated as follows:

[0035]

[0036] The parameters θ of the Q-network are updated using the gradient dθ, and the vector index IndexQ is updated using the updated parameters θ; updating the vector index IndexQ of the Q-network includes:

[0037] For a i (1≤i≤B), Del(IndexQ,a) i ), where Del is the deletion operator, and the input to the deletion operator is the index structure and behavior a. i Delete from the index structure in Let be the embedding vector of line 'a' stored in the index structure; the time complexity of the deletion operator is O(log M);

[0038] For a i (1≤i≤B), connect the updated basis vectors Let the result of the connection be...

[0039] Insert is an insertion operator. The input to the insertion operator is an index structure and the tuple to be inserted. The function of the insertion operator is to insert the tuple to be inserted into the index structure. The time complexity of the insertion operator is O(log M).

[0040] Updating the parameters and vector indices of the target Q-network based on the expected returns includes:

[0041] Every C steps, the parameters in the target Q-network are updated; wherein, the first step: the parameters in the state representation module and the Softmax layer of the Q-network are copied: Step 2: Update the behavior embedding vector, and denote the set of behaviors whose embedding vectors have been updated in the past C steps.

[0042] For each action 'a' in the action space, the following operators are executed sequentially to update the vector index IndexTargetQ of the target Q network:

[0043] Del(IndexTargetQ,a);

[0044] AddrQuery(IndexQ, a) retrieves... The memory address, and thus obtain

[0045] Where AddrQuery is the address lookup operator, the input of which is the index structure and the behavior a, and it returns in O(1) time. The memory address.

[0046] Preferably, the database operators include: index creation, query, address lookup, deletion, and insertion operators.

[0047] Preferably, after obtaining the current environment state and before storing the preset behavior, the environment feedback, the current environment state, and the updated environment state as experience samples in the experience pool, the method further includes:

[0048] Determine whether the number of experience samples in the experience pool is greater than B; if not, return to the step of obtaining the current environment state; if yes, proceed to the step of extracting multiple experience samples from the experience pool and calculating the corresponding expected return based on the database operator; wherein, the step of extracting multiple experience samples from the experience pool and calculating the corresponding expected return based on the database operator includes: extracting B experience samples from the experience pool and calculating the expected return corresponding to each experience sample based on the database operator.

[0049] To address the above problems, this application also provides a Q network control device, comprising:

[0050] A module is established to initialize the parameters of the Q-network and the target Q-network, and to establish the vector indices of the Q-network and the target Q-network.

[0051] The reset module is used to obtain the current environment state. If the current environment state is empty or terminated, the running environment is reset and the initial state of the running environment is obtained as the updated environment state.

[0052] An execution module is used to select and execute a preset behavior, obtain environmental feedback and the updated environmental state; wherein, the preset behavior is a behavior randomly selected in behavior space A or the optimal behavior selected in behavior space A using database operators;

[0053] The storage module is used to store the preset behavior, the environmental feedback, the current environmental state, and the updated environmental state as experience samples in the experience pool; wherein, the experience pool is a storage unit for storing each of the experience samples;

[0054] An extraction module is used to extract multiple experience samples from the experience pool and calculate the corresponding expected returns based on the database operator; wherein the database operator is an operator implemented based on the approximate nearest neighbor search technique;

[0055] An update module is used to update the parameters and vector indexes of the Q network and the parameters and vector indexes of the target Q network according to the expected returns.

[0056] To address the aforementioned problems, this application also provides a Q network control device, including a memory for storing computer programs;

[0057] A processor is used to implement the steps of the Q-network control method as described above when executing the computer program.

[0058] To address the aforementioned problems, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the Q-network control method described above.

[0059] The Q-network control method provided in this application initializes the parameters of both the Q-network and the target Q-network, establishing vector indices for both. Then, it obtains the current environment state. If the current environment state is empty or terminated, it resets the operating environment and obtains the initial state as the current environment state. Next, it selects and executes a preset behavior, obtaining environmental feedback and the updated environment state. The preset behavior is either a randomly selected behavior in behavior space A or the optimal behavior selected in behavior space A using a database operator. The preset behavior, environmental feedback, current environment state, and updated environment state are then stored as experience samples in an experience pool. The experience pool is a storage unit for each experience sample. Multiple experience samples are extracted from the experience pool, and the corresponding expected returns are calculated based on the database operator. Finally, the parameters and vector indices of both the Q-network and the target Q-network are updated based on the expected returns. The database operator is an operator implemented using approximate nearest neighbor search technology. Compared with previous training control methods, this scheme introduces approximate nearest neighbor search technology to obtain the database operator, thereby greatly reducing the search range. Compared to the previous DQN which traversed the entire action space, it increases training and decision-making efficiency and can be applied to large-scale discrete behavior scenarios.

[0060] The Q network control device and computer-readable storage medium provided in this application correspond to the Q network control method described above, and have the same beneficial effects. Attached Figure Description

[0061] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 A flowchart illustrating a Q-network control method provided in an embodiment of this application;

[0063] Figure 2 This is a schematic diagram of the Q-network structure in DQN;

[0064] Figure 3 A schematic diagram of an adaptive deep Q-network structure provided in an embodiment of this application;

[0065] Figure 4 A schematic diagram of the training information flow of an adaptive deep Q-network provided in an embodiment of this application;

[0066] Figure 5 A schematic diagram of the decision information flow of an adaptive deep Q-network provided in an embodiment of this application;

[0067] Figure 6 A schematic diagram of a Q-network control device provided in an embodiment of this application;

[0068] Figure 7 This is a structural diagram of a Q-network control device provided in another embodiment of this application. Detailed Implementation

[0069] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0070] The core of this application is to provide a Q-network control method, device, and readable storage medium to solve the problem that DQN is difficult to apply in large-scale discrete behavior scenarios. For example, in recommendation scenarios, the action space consists of all items, and the action space usually reaches tens of millions or even hundreds of millions. If the training efficiency is low, the model will not converge for a long time, which will have a significant impact on business metrics. If the decision-making efficiency is low, it will be difficult to meet the real-time requirements of the recommendation system.

[0071] The training and decision-making process proposed in this application relies on the following five database operators:

[0072] BuildIndex: Input key-value pairs Where v a It is a high-dimensional vector. BuildIndex output storage. The index structure supports fast and high-precision approximate queries (see query operators), and the time complexity of BuildIndex is O(M log M).

[0073] Query: Given an index structure and a vector q, find the desired value in the index structure in O(log M) time. Make:

[0074] Address lookup (AddrQuery): Given an index structure and behavior a∈[1, M], return (v) in O(1) time. a The memory address of , a).

[0075] Deletion operator (Del): Given an index structure and behavior a∈[1,M], delete (v) from the index structure. a a). The time complexity of this operator is O(log M).

[0076] Insert operator: Input index structure and the tuple to be inserted. Will Insert an index structure. The time complexity of this operator is O(log M).

[0077] To implement the above operators, approximate nearest neighbor search techniques can be considered, such as the Navigable Small World (NSW) algorithm and the Hierarchical Navigable Small World (HNSW) algorithm.

[0078] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0079] Figure 1 A flowchart of a Q-network control method provided in this application embodiment, applied to a distributed storage system, such as... Figure 1 As shown, the method includes:

[0080] S10: Initialize the parameters of the Q-network and the target Q-network, and establish the vector indices of the Q-network and the target Q-network.

[0081] In this embodiment, there are no restrictions on the specific method for initializing the parameters of the Q-network and the target Q-network, nor are there any restrictions on the specific method for establishing the vector indices of the Q-network and the target Q-network. Furthermore, there are no specific restrictions on the environment of the Q-network itself in this embodiment.

[0082] S11: Get the current environment state. If the current environment state is empty or terminated, reset the runtime environment and get the initial state of the runtime environment as the updated environment state.

[0083] S12: Select and execute preset behaviors to obtain environmental feedback and updated environmental status.

[0084] The preset behavior is either a behavior randomly selected in behavior space A or the optimal behavior selected in behavior space A using database operators. The specific choice between random selection and selection using database operators can be determined by setting preset probabilities.

[0085] S13: Store the preset behavior, environmental feedback, current environmental state, and updated environmental state as experience samples in the experience pool.

[0086] The experience pool is a storage unit for storing each experience sample. The elements in each experience sample are determined according to the preset behavior and the corresponding environmental feedback, initial state and current environmental state. Therefore, the elements in two sets of experience samples may be the same or different. It should be noted that in this embodiment, the number of sets of experience samples in the experience pool is not limited.

[0087] S14: Extract multiple experience samples from the experience pool and calculate the corresponding expected returns based on database operators.

[0088] Database operators are operators implemented based on the approximate nearest neighbor search technique. Here, "operator" is another name for database operations.

[0089] S15: Update the parameters and vector indices of the Q network and the target Q network according to the expected return.

[0090] In this embodiment, the specific parameter acquisition method for the above steps, including but not limited to preset behavior, environmental feedback, initial state, and environment, is not specifically limited. This application specifically addresses two problems in DQN: (1) DQN has limited expressive power. Specifically, DQN cannot establish a personalized action embedding space for a state (the physical meaning of the action embedding space is the interest space. Establishing a personalized interest space for different users will achieve better results). (2) DQN has low training and decision efficiency. To address problem (1), this application proposes two key mechanisms: for the same action, a set of basis vectors is introduced; and according to the different states, the weights of each basis vector are adaptively generated, and the weighted sum of the basis vectors is used as the action embedding vector. Under these two mechanisms, the same action has different embedding representations in different states, thereby establishing a personalized action embedding space for each state. To address problem (2), this application introduces a database operator in the training and decision process, reducing the training complexity from O(SBM) to O(SB log M) and the decision complexity from O(M) to O(log M). Therefore, the method proposed in this application achieves better results than DQN in general scenarios and is applicable to scenarios with large-scale action spaces, such as recommendation systems. Furthermore, this application also considers a discrete action space, denoted as M. For convenience, the actions themselves are also used as indexes.

[0091] This application proposes an adaptive deep Q-network algorithm based on database operators, which includes the following key points: (1) It proposes a novel approach to improve the expressive power of DQN: establishing a personalized action embedding space for the state. (2) It proposes an adaptive deep Q-network that implements the above approach, the key point of which is: adaptively adjusting the action embedding vector according to the state. (3) It proposes a training and decision-making process based on database operators. (4) It points out a specific technical route for implementing these database operators: approximate nearest neighbor search. (5) In the training process, the property of local update of the embedding dictionary is used to avoid rebuilding the index. (6) In the training and decision-making process, the argmax operator in DQN is approximated by the query operator. The adaptive Q-network proposed in this application can be applied to various variants of DQN, such as Double DQN, Dueling Network, etc. The training and decision-making process proposed in this application is not limited to the model proposed in this embodiment, and can be applied to more general models. Any model that can transform the calculation process into an inner product form can be used. In addition, for the nonlinear activation function in the calculation process, polynomial approximation and other methods can be considered to expand it and then transform it into an inner product form.

[0092] The Q-network control method provided in this application initializes the parameters of the Q-network and the target Q-network, establishing vector indices for both. Then, it obtains the current environment state. If the current environment state is empty or terminated, it resets the operating environment and obtains the initial state as the current environment state. Next, it selects and executes a preset behavior, obtaining environmental feedback and the updated environment state. The preset behavior is either a randomly selected behavior in behavior space A or the optimal behavior selected in behavior space A using a database operator. The preset behavior, environmental feedback, current environment state, and updated environment state are then stored as experience samples in an experience pool. The experience pool is a storage unit for each experience sample. Multiple experience samples are extracted from the experience pool, and the corresponding expected returns are calculated based on the database operator. Finally, the parameters and vector indices of the Q-network and the target Q-network are updated according to the expected returns. The database operator is an operator implemented based on approximate nearest neighbor search technology. Compared with previous training control methods, this scheme introduces approximate nearest neighbor search technology to obtain the database operator, thereby greatly reducing the search range. Compared to the previous DQN which traversed the entire action space, it increases training and decision-making efficiency and can be applied to large-scale discrete behavior scenarios.

[0093] In Q-learning, the agent adopts behavioral policies π. E It interacts with the environment and uses feedback signals from the environment to optimize its target strategy π. G Specifically, at a certain time step t, the environment is in state st, and the agent, based on π...E Select action a t And it acts on the environment. The environment transitions to a new state s in step t+1. t+1 And provide the agent with corresponding feedback r t+1 Intelligent agents utilize (s t a t r t+1 s t+1 Estimate π G The state-action value function Q(s, a) (meaning that after an agent takes action a in state s, it follows the π-value function) G (the expected return), and using the estimated results, greedily improve π. G Repeat the above steps until convergence (i.e., π). G (No longer changing).

[0094] Figure 2 This is a schematic diagram of the Q-network structure in DQN; for example... Figure 2 As shown, the state representation layer is implemented using a deep neural network (DNN). After the state s is processed by the state representation layer, a low-dimensional dense representation φ(s) is obtained. The embedding layer is a matrix with M columns, where the first ≤ a ≤ M columns correspond to the encoding vector of the a-th behavior. The output layer outputs the state value function Q(s, a) = φ(s) for all M behaviors. T e a The DQN iterative optimization of θ consists of two steps in each iteration: Step 1. Store the samples obtained from the interaction between the behavioral policy πE and the environment into the experience pool; Step 2. Randomly select B samples from the experience pool. Update θ. Specifically, given a sample (s) i a i r i ,s′ i The output of the Q network is Q(s). i a i The goal is to minimize Q(s). i a i ;θ) and The distance between, of which This is a frozen parameter introduced to stabilize the learning process of θ; DQN periodically uses θ to update...

[0095] In this embodiment, the Q-network and the target Q-network adopt adaptive behavior representation. Figure 3 A schematic diagram of an adaptive deep Q-network structure provided in an embodiment of this application; as shown Figure 3As shown, the innovation of the adaptive deep Q-network compared to DQN lies in the adaptive action representation module within the dashed box. The input to this module includes: the representation φ(s) extracted from state s by the state representation layer and the action a. The computation steps of this module are as follows:

[0096] The first step is to extract the basis vectors of action 'a', and store these basis vectors in K embedding dictionaries. Let the dimension of the embedding vectors be D, and let W be the embedding matrix corresponding to the 1≤j≤Kth embedding dictionary. j ∈R D×M Input 'a', get W. j The a-th column vector in the array is denoted as That is, from K embedding matrices W1, W2, ..., W K Extract the a-th column vector from each column as the basis vector for row a. These are the K basis vectors of action a.

[0097] The second step is to determine the weights corresponding to the basis vectors. The weights are calculated in the Softmax layer. Let the learnable parameters in the Softmax layer be... Input φ(s), and calculate the output as a distribution according to the following formula.

[0098]

[0099] Where u1, ..., u K Here, φ(s) is a learnable parameter, and φ(s) is the output of the state representation module.

[0100] The third step is to calculate the representation vector of action a in state s:

[0101]

[0102] At this point, the representation of 'a' is related to state 's', and the state-action value function Q(s, a) is the output function of the Q-network, Q(s, a) = φ. T (s)e a (s).

[0103] The above embodiments propose an adaptive deep Q-network. Based on the aforementioned database operator, the training process of the improved DQN in this application embodiment is as follows: Figure 4 This application provides a schematic diagram of the training information flow of an adaptive deep Q-network as an embodiment of the present application; for example... Figure 4 As shown, the specific steps include the following:

[0104] Step 0. Initialize the experience pool as a queue with a capacity of BufferSize.

[0105] Step 1. Parameter initialization.

[0106] Step 2. Build an index.

[0107] Step 3. Rounds 1, …, EpisodeNum:

[0108] Step 3.1 Reset the environment and observe the initial state s;

[0109] Step 3.2 If s is a terminal state, return to Step 3;

[0110] Step 3.3 Set a preset probability ∈, that is, with a probability of ∈, randomly select an action a from the action space, otherwise use the database operator to select the optimal action in the action space. The specific scheme of using the database operator is as follows:

[0111] Step 3.3.1 Construct the query vector q T = [p1φ(s) T ,..., p k φ(s) T ;

[0112] Step 3.3.2 Retrieve the approximately optimal action: -, a = Query(IndexQ, q);

[0113] Step 3.4 Execute the action a and observe the feedback r of the environment and the new state s';

[0114] Step 3.5 Put the experience (s, a, r, s′) into the experience pool D; let s ← s′;

[0115] Step 3.6 If |D| < B, return to Step 3.2;

[0116] Step 3.7 Randomly sample B experience samples from D According to the experience samples (s i , a i , r i , s′ i ) calculate the expected return y at the state-action (s i , a i ) i .

[0117] Step 3.8 Update the Q network.

[0118] Step 3.9 Update the target Q network.

[0119] Step 3.10 Return to Step 3.2

[0120] From the above formula, we can see that a0 is the near-optimal behavior, and the time complexity of retrieving the near-optimal behavior is O(log M), which is significantly more efficient than the O(M) complexity of accurately calculating the optimal behavior in DQN. Therefore, according to the training method in this application, the complexity is O(SB log M).

[0121] In Steps 1 and 2 above, the parameters of the Q-network and the target Q-network are initialized, and the vector indices of the Q-network and the target Q-network are established, including:

[0122] Randomly initialize the parameters θ of the Q network = (θ s θ a W1, ..., W K ), where θ s θ is a parameter in the state representation module. a These are the parameters in the Softmax layer, W1, ..., W K These are the K embedding matrices in the behavior representation module. The target Q-network and the Q-network have the same network structure, and their parameters are initialized to...

[0123] The vector index IndexQ of the Q network and the vector index IndexTargetQ of the target Q network are established using the following steps;

[0124] For any action a, connect a on W1, ..., W K The corresponding basis vectors are obtained

[0125]

[0126] Connect a in The corresponding basis vectors are obtained

[0127]

[0128] The process of establishing the vector indices IndexQ and IndexTargetQ is as follows:

[0129]

[0130]

[0131] Where M is the size of the behavior space, the input of BuildIndex is all key-value pairs, and the output is a vector index that stores the input. This vector index can support fast and high-precision approximate queries. The time complexity of BuildIndex is O(Mlog M).

[0132] Additionally, as in Step 3.6, after obtaining the current environment state and before storing the preset behavior, environment feedback, current environment state, and updated environment state as experience samples in the experience pool, it can be determined whether the number of experience samples in the experience pool is greater than B. If the number of experience samples is not greater than B, then return to the step of obtaining the current environment state (in the above, this is returning to Step 3.2; as long as the requirements are met, no specific requirements are specified here); if the number of experience samples is greater than B, then extract B experience samples from the experience pool, and calculate the expected benefit corresponding to each experience sample based on the database operator. Through the above scheme, extraction is only performed when the number of experience groups meets the preset number B; otherwise, experience group generation will continue, thus ensuring the integrity of the scheme and ensuring that the number of experience groups meets the requirements.

[0133] Step 3.7 is based on the empirical sample (s) i a i r i ,s′ i ) computed state-behavior(s) i a i Expected return y under ) i As follows, where s i As the initial state, a i For random actions, r i For environmental feedback, s′ i This represents a random environment state.

[0134] If s′ i If it is a terminated state, then y i =r i ;

[0135] Otherwise, in the target Q-network, the state representation is computed. Weights of basis vectors Construct the query vector:

[0136]

[0137] The query results are as follows:

[0138]

[0139] Where Query is the query operator, and the input to the query operator is the index structure and the query vector. If the size of the index structure is M, then the query operator can find the target in O(logM) time. Make: get Afterwards, the expected return y i The calculation is as follows:

[0140]

[0141] Steps 3.8 and 3.9 update the parameters and vector indices of the Q-network and the target Q-network based on the expected return, including the following steps:

[0142] The gradient dθ of parameter θ is calculated as follows, and parameter θ is updated using dθ.

[0143]

[0144] Then, update the vector index IndexQ using the updated parameter θ.

[0145] Updating the vector index IndexQ involves the following steps:

[0146] For each a i (1≤i≤B), Del(IndexQ,a) i );

[0147] For each a i (1≤i≤B), connect the updated basis vectors Let the result of the connection be...

[0148] Insert is an insertion operator that takes an index structure and the tuple to be inserted as inputs and inserts the tuple into the index structure. The time complexity of this operator is O(log M).

[0149] The steps to update the target Q network are as follows:

[0150] Update the parameters in the target Q network every C steps.

[0151] Step 1: Copy the parameters from the state representation module and the Softmax layer in the Q network:

[0152] Step 2: Update the behavior embedding vector.

[0153] The set of behaviors that update the embedding vector in the past C steps.

[0154] For each action 'a' in A, execute the following operators in sequence:

[0155] Del(IndexTargetQ,a);

[0156] AddrQuery(IndexQ, a) retrieves... The memory address, and thus obtain

[0157] Update the vector index IndexTargetQ of the target Q network.

[0158] Here, AddrQuery is the address lookup operator, whose input is the index structure and the behavior a∈[1,M], and returns a value in O(1) time. The memory address.

[0159] Considering the constraints on the decision-making process, the preferred solution is provided here. Figure 5 This application provides a schematic diagram of the decision information flow of an adaptive deep Q-network, as shown in the embodiments of this application. Figure 5 As shown, similarly to the above embodiments, after training, compared to DQN, the complexity of the decision-making process in this application decreases from O(M) to O(log M). The decision-making process is as follows:

[0160] Step 10. Given state s, construct query vector q T =[p1φ(s)] T , ..., p k φ(s) T ];

[0161] Step 11._, a=Query(IndexQ, q)

[0162] Step 12. Return to a.

[0163] In the above embodiments, the Q-network control method has been described in detail. This application also provides embodiments corresponding to the Q-network control device. It should be noted that this application describes the embodiments of the device part from two perspectives: one is based on the functional modules, and the other is based on the hardware.

[0164] Figure 6 This is a schematic diagram of a Q-network control device provided in an embodiment of this application. The device includes:

[0165] Module 10 is established to initialize the parameters of the Q-network and the target Q-network, and to establish the vector indices of the Q-network and the target Q-network.

[0166] The reset module 11 is used to obtain the current environment state. If the current environment state is empty or terminated, the running environment is reset and the initial state of the running environment is obtained as the updated environment state.

[0167] Execution module 12 is used to select and execute preset behaviors, obtain environmental feedback and updated environmental status; wherein, the preset behavior is a behavior randomly selected in behavior space A or the optimal behavior selected in behavior space A using database operators;

[0168] Storage module 13 is used to store preset behaviors, environmental feedback, current environmental state and updated environmental state as experience samples in the experience pool; wherein, the experience pool is a storage unit for storing each experience sample.

[0169] Extraction module 14 is used to extract multiple experience samples from the experience pool and calculate the corresponding expected returns based on the database operator; wherein, the database operator is an operator implemented based on the approximate nearest neighbor search technique;

[0170] Update module 15 is used to update the parameters and vector indices of the Q network and the parameters and vector indices of the target Q network according to the expected returns.

[0171] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, the embodiments of the apparatus and the embodiments of the method described herein will not be repeated here.

[0172] The Q-network control device provided in this application corresponds to the above method and therefore has the same beneficial effects as the above method.

[0173] Figure 7 A structural diagram of a Q-network control device provided in another embodiment of this application is shown below. Figure 7 As shown, the Q network control device includes: a memory 20 for storing computer programs;

[0174] The processor 21 is used to implement the steps of the Q-network control method mentioned in the above embodiments when executing a computer program.

[0175] The Q network control device provided in this embodiment may include, but is not limited to, smartphones, tablets, laptops, or desktop computers.

[0176] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one hardware form selected from Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0177] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 20 is used to store at least the following computer program 201, which, after being loaded and executed by the processor 21, can implement the relevant steps of the Q-network control method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, and the storage method may be temporary or permanent storage. The operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include, but is not limited to, the data involved in the aforementioned Q-network control method.

[0178] In some embodiments, the Q network control device may further include a display screen 22, an input / output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.

[0179] Those skilled in the art will understand that the structure shown in the figure does not constitute a limitation on the Q-network control device and may include more or fewer components than shown.

[0180] The Q-network control device provided in this application includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the following method: the Q-network control method involved in the above embodiments.

[0181] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus and their corresponding beneficial effects, which will not be repeated here.

[0182] The Q-network control device provided in this application corresponds to the above method and therefore has the same beneficial effects as the above method.

[0183] Finally, this application also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps described in the above method embodiments.

[0184] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0185] Since the embodiments of the readable storage medium portion correspond to the embodiments of the method portion, the embodiments of the apparatus portion and their corresponding beneficial effects are described in the description of the embodiments of the method portion, and will not be repeated here.

[0186] The computer-readable storage medium provided in this embodiment corresponds to the method described above, and therefore has the same beneficial effects as the method described above.

[0187] The foregoing provides a detailed description of a Q-network control method, apparatus, and readable storage medium provided in this application. The various embodiments are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0188] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the aforementioned element.

Claims

1. A Q-network control method, characterized in that, Suitable for product recommendation scenarios, including: The parameters of the Q-network and the target Q-network are initialized, and vector indices for the Q-network and the target Q-network are established. The Q-network and the target Q-network adopt adaptive behavior representations. For the same product, a set of basis vectors is introduced. According to the different states, the weights of each basis vector are generated, and the weighted sum of the basis vectors is used as the embedding vector of the product. The same product has different embedding representations in different states, and an action embedding space is established for each user. The physical meaning of the action embedding space is the interest space. Obtain the current environment state. If the current environment state is empty or terminated, reset the running environment and obtain the initial state of the running environment as the updated environment state. Select and execute a preset behavior to obtain environmental feedback and the updated environmental state; wherein, the preset behavior is a behavior randomly selected in behavior space A or the optimal behavior selected in behavior space A using database operators; the action space consists of all items, and the size of the action space is... Action space ,Behavior ;Use database operators to select the optimal behavior in the behavior space without traversing the entire action space; The preset behavior, the environmental feedback, the current environmental state, and the updated environmental state are stored as experience samples in the experience pool; wherein, the experience pool is a storage unit for storing each of the experience samples. Multiple experience samples are extracted from the experience pool, and the corresponding expected returns are calculated based on the database operator; wherein, the database operator is an operator implemented based on the approximate nearest neighbor search technique; Update the parameters and vector indexes of the Q network and the parameters and vector indexes of the target Q network according to the expected returns.

2. The Q-network control method according to claim 1, characterized in that, The initialization of the parameters of the Q-network and the target Q-network, and the establishment of the vector indices of the Q-network and the target Q-network, include: Randomly initialize the parameters of the Q network. ,in, These are parameters in the state representation module. These are parameters in the Softmax layer. The behavior representation module contains K embedding matrices, and the target Q-network and the Q-network have the same network structure, with parameters initialized to... ; The vector index IndexQ of the Q network and the vector index IndexTargetQ of the target Q network are established using the following steps; For any action a, connection a is in The corresponding basis vectors are obtained : ； Connect a in The corresponding basis vectors are obtained : ； The vector indices IndexQ and IndexTargetQ are established as follows: IndexQ = BuildIndex( ); IndexTargetQ = BuildIndex( ); Where M is the size of the behavior space, BuildIndex takes all key-value pairs as input and outputs a vector index that stores the inputs, and BuildIndex has a time complexity of O(MlogM).

3. The Q-network control method according to claim 2, characterized in that, The step of extracting multiple experience samples from the experience pool and calculating the corresponding expected returns based on the database operators includes: Based on empirical samples Calculate state-behavior Expected return y i ; where s i For the current state, a i For the preset behavior, r i For the environmental feedback, The updated environmental state; like If it is a terminated state, then y i =r i Otherwise: In the target Q-network, calculate the state representation. The weights of the basis vectors Construct the query vector: ； The query results are as follows: =Query(IndexTargetQ, ) Wherein, Query is the query operator, and the input of the query operator is the index structure and the query vector. If the size of the index structure is M, then the query operator finds the target in O(logM) time. , so that: ,get Afterwards, the expected return y i The calculation is as follows: 。 4. The Q-network control method according to claim 3, characterized in that, Updating the parameters and vector indices of the Q network based on the expected returns includes: The parameters of the Q-network are calculated as follows: gradient : ； Using the gradient Update the parameters of the Q network. and using the updated parameters Update the vector index IndexQ; updating the vector index IndexQ of the Q network includes: For a i (1≤i≤B), Del(IndexQ,a i ), where Del is the deletion operator, and the input to the deletion operator is the index structure and behavior a. i Delete from the index structure ,in Let be the embedding vector of 'a' stored in the index structure; the time complexity of the delete operator is O(n log n). ; For a i (1≤i≤B), connect the updated basis vectors Let the result of the connection be . ; Insert(IndexQ, ( , a i ) ); where Insert is the insertion operator, the input of which is an index structure and the tuple to be inserted, and the function of the insertion operator is to insert the tuple to be inserted into the index structure. The time complexity of the insertion operator is O(n log n). ; Updating the parameters and vector indices of the target Q-network based on the expected returns includes: Every C steps, the parameters in the target Q-network are updated; wherein, the first step: the parameters in the state representation module and the Softmax layer of the Q-network are copied: Step 2: Update the behavior embedding vector, and denote the set of behaviors whose embedding vectors have been updated in the past C steps. ; For each action a in A, the following operators are executed sequentially to update the vector index IndexTargetQ of the target Q network: Del(IndexTargetQ,a); AddrQuery(IndexQ, a) retrieves... The memory address, and thus obtain ; Insert(IndexTargetQ, ); where AddrQuery is the address lookup operator, the input of which is the index structure and the behavior a, and returns in O(1) time. The memory address.

5. The Q-network control method according to any one of claims 1 to 4, characterized in that, The database operators include: index creation, query, address lookup, deletion, and insertion operators.

6. The Q-network control method according to claim 5, characterized in that, After obtaining the current environment state, and before storing the preset behavior, the environment feedback, the current environment state, and the updated environment state as experience samples in the experience pool, the method further includes: Determine whether the number of experience samples in the experience pool is greater than B; if not, return to the step of obtaining the current environment state; if yes, proceed to the step of extracting multiple experience samples from the experience pool and calculating the corresponding expected return based on the database operator; wherein, the step of extracting multiple experience samples from the experience pool and calculating the corresponding expected return based on the database operator includes: extracting B experience samples from the experience pool and calculating the expected return corresponding to each experience sample based on the database operator.

7. A Q-network control device, characterized in that, Suitable for product recommendation scenarios, including: A module is established to initialize the parameters of the Q-network and the target Q-network, and to establish the vector indices of the Q-network and the target Q-network. The Q-network and the target Q-network adopt adaptive behavioral representations. For the same product, a set of basis vectors is introduced. The weights of each basis vector are generated according to the different states, and the weighted sum of the basis vectors is used as the embedding vector of the product. The same product has different embedding representations in different states, and an action embedding space is established for each user. The physical meaning of the action embedding space is the interest space. The reset module is used to obtain the current environment state. If the current environment state is empty or terminated, the running environment is reset and the initial state of the running environment is obtained as the updated environment state. The execution module is used to select and execute preset behaviors, obtain environmental feedback and the updated environmental state; wherein, the preset behavior is a behavior randomly selected in behavior space A or the optimal behavior selected in behavior space A using database operators; the action space consists of all goods, and the size of the action space is... Action space ,Behavior ;Use database operators to select the optimal behavior in the behavior space without traversing the entire action space; The storage module is used to store the preset behavior, the environmental feedback, the current environmental state, and the updated environmental state as experience samples in the experience pool; wherein, the experience pool is a storage unit for storing each of the experience samples; An extraction module is used to extract multiple experience samples from the experience pool and calculate the corresponding expected returns based on the database operator; wherein the database operator is an operator implemented based on the approximate nearest neighbor search technique; An update module is used to update the parameters and vector indexes of the Q network and the parameters and vector indexes of the target Q network according to the expected returns.

8. A Q-network control device, characterized in that, Includes memory used to store computer programs; A processor, configured to implement the steps of the Q-network control method as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the Q-network control method as described in any one of claims 1 to 6.

Images