Training methods, devices, and products for request filtering models

Abstract

This disclosure provides a training method, apparatus, electronic device, storage medium, and program product for a request filtering model, relating to the field of artificial intelligence technology, specifically deep reinforcement learning and code recommendation. The specific implementation scheme is as follows: During the code recommendation process, a training sample set is generated by using a deep Q-network to be trained, employing a greedy strategy to accumulate experience regarding whether to filter code recommendation requests. Each training sample includes first state information, response action, reward information, and second state information. A target deep Q-network is then used to determine a target Q-value based on randomly selected training samples from the training sample set. The training deep Q-network is updated based on the loss between the actual Q-value and the target Q-value corresponding to the same training sample in the training sample set, ultimately obtaining the request filtering model. This disclosure can accurately filter inferior code recommendation requests, improving the efficiency of the code recommendation process.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to the fields of deep reinforcement learning and code recommendation, and particularly to training methods and apparatus for request filtering models, as well as request filtering methods, apparatus, electronic devices, storage media, and computer program products, which can be used in code recommendation scenarios. Background Technology

[0002] During code development within an IDE (Integrated Development Environment), users can trigger code recommendation operations based on a code recommendation model when they pause within the IDE. Specifically, when pausing within the IDE, a code recommendation request is sent to the backend code recommendation model to retrieve recommended code generated by the model. In actual code recommendation processes, a large number of low-quality code recommendation requests can degrade the output quality of the code recommendation model, significantly increasing the cost of invoking it. Summary of the Invention

[0003] This disclosure provides a method and apparatus for training a request filtering model, as well as a request filtering method, apparatus, electronic device, storage medium, and computer program product.

[0004] According to the first aspect, a training method for a request filtering model is provided, comprising: during the code recommendation process, accumulating experience on whether to filter code recommendation requests using a greedy strategy through a deep Q-network to be trained, generating a training sample set, wherein each training sample in the training sample set includes first state information, a response action representing whether to filter code recommendation requests, reward information based on the response action, and second state information, wherein the first state information represents the environmental state of the code development process, the second state information represents the first state information after being changed based on the response action, the response action is determined by the actual Q-value output by the deep Q-network to be trained according to the first state information, and the actual Q-value represents the cumulative expected reward for the response action under the first state information; determining a target Q-value through a target deep Q-network based on training samples randomly selected from the training sample set, wherein the target Q-value represents the maximum expected cumulative reward for the response action in the selected training samples; updating the deep Q-network to be trained according to the loss between the actual Q-value and the target Q-value corresponding to the same training sample in the training sample set, so as to finally obtain the request filtering model.

[0005] According to the second aspect, a request filtering method is provided, including: obtaining code recommendation requests; and determining a response action representing whether to filter code recommendation requests using a trained request filtering model, wherein the request filtering model is trained using any implementation of the first aspect.

[0006] According to a third aspect, a training apparatus for a request filtering model is provided, comprising: an accumulation unit configured to, during the code recommendation process, accumulate experience on whether to filter code recommendation requests using a greedy strategy through a deep Q-network to be trained, generating a training sample set, wherein each training sample in the training sample set includes first state information, a response action representing whether to filter code recommendation requests, reward information based on the response action, and second state information, wherein the first state information represents the environmental state of the code development process, the second state information represents the first state information changed based on the response action, the response action is determined by the actual Q-value output by the deep Q-network to be trained based on the first state information, and the actual Q-value represents the cumulative expected reward for the response action under the first state information; a first determination unit configured to, through a target deep Q-network, determine a target Q-value based on training samples randomly selected from the training sample set, wherein the target Q-value represents the maximum expected cumulative reward for the response action in the selected training samples; and an update unit configured to, based on the loss between the actual Q-value and the target Q-value corresponding to the same training sample in the training sample set, update the deep Q-network to be trained to finally obtain the request filtering model.

[0007] According to the fourth aspect, a request filtering method is provided, comprising: an acquisition unit configured to acquire code recommendation requests; and a second determination unit configured to determine, through a trained request filtering model, a response action characterizing whether to filter the code recommendation request, wherein the request filtering model is trained using any implementation of the third aspect.

[0008] According to a fifth aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method as described in any implementation of the first or second aspect.

[0009] According to a sixth aspect, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing a computer to perform the method described in any implementation of the first or second aspect.

[0010] According to the seventh aspect, a computer program product is provided, comprising: a computer program that, when executed by a processor, implements the method as described in either the first or second aspect.

[0011] According to the technology disclosed herein, a training method for a request filtering model is provided. This method employs a DDQN (Double Deep Q-network) architecture based on a training deep Q-network and a target deep Q-network. During the code recommendation process, the training deep Q-network uses a greedy strategy to accumulate experience regarding whether to filter code recommendation requests. The training deep Q-network is then updated based on the loss between the actual Q-value obtained from the training deep Q-network and the target Q-value obtained from the target deep Q-network. This results in a request filtering model with the function of filtering code recommendation requests, which can accurately filter low-quality code recommendation requests and improve the efficiency and effectiveness of the code recommendation process.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0014] Figure 1 This is an exemplary system architecture diagram that can be applied to an embodiment of this disclosure;

[0015] Figure 2 This is a flowchart of an embodiment of a request filtering model training method according to this disclosure;

[0016] Figure 3 This is a schematic diagram illustrating an application scenario of the training method for the request filtering model according to this embodiment;

[0017] Figure 4 This is a flowchart of one embodiment of the method for determining first state information according to this embodiment;

[0018] Figure 5 This is a flowchart of one embodiment of the method for determining reward information according to this embodiment;

[0019] Figure 6 This is a schematic diagram of one embodiment of the training process of the request filtering model according to this disclosure;

[0020] Figure 7 This is a flowchart of one embodiment of the request filtering method according to this disclosure;

[0021] Figure 8 This is a structural diagram of one embodiment of a training apparatus for a request filtering model according to the present disclosure;

[0022] Figure 9This is a structural diagram of one embodiment of the request filtering device according to the present disclosure;

[0023] Figure 10 This is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure. Detailed Implementation

[0024] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0025] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0026] Figure 1 An exemplary architecture 100 of a training method and apparatus for the request filtering model of this disclosure, and a request filtering method and apparatus are shown.

[0027] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. The communication connections between terminal devices 101, 102, and 103 form a network topology. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0028] Terminal devices 101, 102, and 103 can be hardware or software that supports network connectivity for data interaction and processing with the server. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices supporting network connectivity, information acquisition, interaction, display, and processing functions, including but not limited to smartphones, tablets, e-book readers, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. They can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are imposed here.

[0029] Server 105 can be a server providing various services. For example, in the code recommendation process using code recommendation requests from terminal devices 101, 102, and 103, a DDQN network architecture based on a training deep Q-network and a target deep Q-network is employed. During the code recommendation process, the training deep Q-network uses a greedy strategy to accumulate experience regarding whether to filter code recommendation requests. The training deep Q-network is updated based on the loss between the actual Q-value obtained from the training deep Q-network and the target Q-value obtained from the target deep Q-network, ultimately resulting in a background processing server for the request filtering model. Alternatively, after obtaining the trained request filtering model, the background processing server determines whether to filter code recommendation requests from terminal devices 101, 102, and 103. As an example, server 105 could also be a cloud server.

[0030] It should be noted that a server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (such as software programs or software modules used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0031] It should also be noted that the training method and request filtering method of the request filtering model provided in the embodiments of this disclosure can be executed by a server, by a terminal device, or by a combination of both. Accordingly, the various parts (e.g., units) included in the training device and request filtering device of the request filtering model can be entirely located in the server, entirely located in the terminal device, or separately located in the server and the terminal device.

[0032] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included. When the training method for the request filtering model and the electronic devices on which it runs do not require data transmission with other electronic devices, the system architecture may only include the training method for the request filtering model and the electronic devices (e.g., servers or terminal devices) on which it runs.

[0033] Please refer to Figure 2 , Figure 2 A flowchart of a training method for a request filtering model provided in this disclosure embodiment, wherein process 200 includes the following steps:

[0034] Step 201: In the code recommendation process, a training sample set is generated by using a greedy strategy to accumulate experience on whether to filter code recommendation requests through a deep Q-network to be trained.

[0035] In this embodiment, the execution entity of the training method for the request filtering model (e.g., Figure 1 The terminal device or server in the process can accumulate experience on whether to filter code recommendation requests by using a greedy strategy through the deep Q-network to be trained during the code recommendation process, and generate a training sample set.

[0036] Each training sample in the training sample set includes first state information, a response action representing whether to filter code recommendation requests, reward information based on the response action, and second state information. The first state information represents the environmental state of the code development process, and the second state information represents the first state information after it has changed based on the response action. The response action is determined by the actual Q value output by the deep Q network to be trained based on the first state information. The actual Q value represents the cumulative expected reward for the response action under the first state information.

[0037] As an example, each training sample in the training sample set can be represented as (S t a t R t S t+1 ), where S t Indicates the first state information, a t Indicates the response action, R t S indicates reward information. t+1 This indicates the second state information.

[0038] Specifically, in the code recommendation process, at each time step, the current environment state (first state information) S, represented by the code recommendation request, will be used. t Input the deep Q-network to be trained. The deep Q-network will observe the current state information and calculate the actual Q value Q(S) of each possible response action. t a t The system uses a greedy strategy to select a response action (filtering code recommendation requests or receiving code recommendation requests) with the highest cumulative expected reward. Then, the deep Q-network to be trained sends the determined response action to the current environment state (first state information) so that the response action interacts with the environment to obtain the changed environment (second state information). The reward information is then calculated based on the response action.

[0039] When the response action is a code recommendation filter request, the code recommendation model cannot receive the code recommendation request. The interaction between the response action and the environment is mainly reflected in continuing to receive the user's recommendation operations in the IDE. When the response action is to receive a code recommendation request, the code recommendation model can receive the code recommendation request, generate and provide recommended code to the user. The interaction between the response action and the environment is mainly reflected in the user's reaction to the recommended code.

[0040] For each time step, its corresponding first state information, response action, reward information, and second state information are stored as a training sample in the experience pool for experience accumulation, thus obtaining a training sample set.

[0041] Cumulative reward expectation refers to the expected value of the total reward obtained after a series of actions and state transitions starting from the current time step in a sequential decision-making process.

[0042] In deep reinforcement learning, rewards are typically signals returned from the environment that guide the deep Q-network to take the correct actions, thereby maximizing the expected cumulative reward. The method of reward calculation can be determined based on the specific scenario and task to improve the performance and stability of the deep Q-network.

[0043] Generally, rewards in code recommendation scenarios should have the following characteristics: Rewards should be relevant to the goal of the recommendation task (e.g., getting users to adopt recommended code as much as possible); rewards should encourage the deep Q-network to complete the task as quickly as possible, meaning rewards should peak upon task completion to incentivize the deep Q-network to learn faster; rewards should avoid excessive sparsity; if rewards only appear at certain specific steps, the deep Q-network may struggle to find the correct action path and perform poorly in real-world tasks; rewards should avoid too many negative impacts; even if the deep Q-network fails, it should still receive some positive rewards to encourage it to try more possible solutions.

[0044] In some examples, the aforementioned agent can employ an ε-greedy strategy to perform the experience accumulation process. The ε-greedy strategy is based on a greedy strategy, selecting the action with the highest actual Q-value in most cases, but also randomly selecting an action with a probability of ε.

[0045] Specifically, the ε-greedy strategy includes the following steps: (1) Initialization: Set an initial value for ε and initialize the deep Q-network. (2) State representation: For the current state, use the deep Q-network to calculate the actual Q-value for each action. (3) Action selection: Select the action with the highest actual Q-value with a probability of 1-ε, and randomly select an action with a probability of ε (usually by uniformly sampling from the action space). (4) Execute the action: Execute the selected action and observe the new state and reward. (5) Update the deep Q-network: Update the weights of the deep Q-network using the current state, action, reward, and new state. (6) Reduce ε: Gradually reduce the value of ε so that the deep Q-network becomes increasingly inclined to select the optimal action.

[0046] The ε-greedy strategy balances exploration and exploitation, avoiding the dilemma of choosing between overexploration and overexploitation. Early in training, ε is typically high, allowing the deep Q-network to explore more states and actions. As training progresses, ε gradually decreases, enabling the network to increasingly favor the selection of optimal actions.

[0047] Step 202: Determine the target Q value using a target depth Q-network based on training samples randomly selected from the training sample set.

[0048] In this embodiment, the aforementioned execution entity can determine the target Q-value using a target depth Q-network based on training samples randomly selected from the training sample set. The target Q-value represents the maximum expected cumulative reward of the response action in the selected training samples.

[0049] As an example, the aforementioned execution entity can iteratively perform the following operations: First, randomly select training samples from the training sample set and then delete the selected training samples from the training sample set; then, calculate the target Q value using the following formula:

[0050] Target Q = r + γmax(Q(s) t+1 a t+1 ))

[0051] Where Target Q represents the target Q value; r represents the reward information obtained after performing the response action a; γ is a discount factor between 0 and 1, used to balance the importance of current rewards and future rewards; max(Q(s) t+1 a t+1 )) indicates the second state information s t+1 Choose the action a that maximizes the Q value from all available actions. t+1 The corresponding Q value.

[0052] Step 203: Update the deep Q-network to be trained based on the loss between the actual Q-value and the target Q-value corresponding to the same training sample in the training sample set, so as to finally obtain the request filtering model.

[0053] In this embodiment, the aforementioned execution entity can update the Q-network to be trained based on the loss between the actual Q-value and the target Q-value corresponding to the same training sample in the training sample set, so as to finally obtain the request filtering model.

[0054] As an example, the aforementioned execution entity can use loss functions such as cross-entropy loss, KL divergence loss, and mean squared error to calculate the loss between the actual Q value and the target Q value. Taking the mean squared error loss function as an example, the aforementioned execution entity can calculate the loss between the actual Q value and the target Q value using the following formula:

[0055] Loss=(Q(s t a t )-(r+γmax(Q(s t+1 a t+1 )))) 2

[0056] Where Q(s) t a t ) represents the actual Q value, (r+γmax(Q(s)) t+1 a t+1 ))) represents the target Q value.

[0057] In this embodiment, the aforementioned execution entity can iteratively execute the following training operations:

[0058] First, calculate the loss between the actual Q-value and the target Q-value for the same training model. Then, update the network weights of the Q-network to be trained using the backpropagation algorithm based on the loss. Specifically, stochastic gradient descent (SGD) or other optimizers can be used to minimize the loss and update the network parameters. For example, the partial derivatives of the loss function with respect to each parameter in the network can be calculated, and then updated according to the rules of SGD or other optimizers.

[0059] Upon reaching a preset termination condition, the trained Q-network of untrained depth is designated as the request filtering model. The preset termination condition could be, for example, training time exceeding a preset time threshold, the number of training iterations exceeding a preset number of iterations threshold, or the training loss converging.

[0060] See also Figure 3 , Figure 3 This is a schematic diagram 300 illustrating an application scenario of the training method for the request filtering model according to this embodiment. Figure 3In the application scenario, the server executing the request filtering model training method deploys a DDQN (Double Deep Q-network) network based on a training-to-be-trained deep Q-network 301 and a target deep Q-network 302, as well as a code recommendation model 303. Users interact with the server during the code development process on their respective terminal devices. Specifically, in the code recommendation process, the server first uses the training-to-be-trained deep Q-network 301 to accumulate experience on whether to filter code recommendation requests issued by terminal devices using a greedy strategy, generating a training sample set 304. Each training sample in the training sample set includes first state information S. t , Indicates whether the response action of the code recommendation request is filtered. t Reward information R based on response actions t Second state information S t+1 The first state information represents the environmental state of the code development process, and the second state information represents the first state information after it changes based on the response action. The response action is determined by the actual Q value output by the deep Q network to be trained based on the first state information. The actual Q value represents the cumulative expected reward for the response action under the first state information. Then, the target deep Q network determines the target Q value based on training samples randomly selected from the training sample set 304. The target Q value represents the maximum expected cumulative reward for the response action in the selected training samples. The deep Q network to be trained is updated based on the loss between the actual Q value and the target Q value corresponding to the same training sample in the training sample set, so as to finally obtain the request filtering model 305.

[0061] This embodiment provides a training method for a request filtering model. It employs a DDQN (Double Deep Q-network) architecture based on a training deep Q-network and a target deep Q-network. During the code recommendation process, the training deep Q-network uses a greedy strategy to accumulate experience regarding whether to filter code recommendation requests. The training deep Q-network is updated based on the loss between the actual Q-value obtained from the training deep Q-network and the target Q-value obtained from the target deep Q-network. This results in a request filtering model with the function of filtering code recommendation requests, which can accurately filter low-quality code recommendation requests and improve the efficiency and effectiveness of the code recommendation process.

[0062] In some optional implementations of this embodiment, the execution entity can perform step 202 as follows:

[0063] First, during the training of the deep Q-network to be trained, in response to determining that after each preset number of iterations, the target deep Q-network is updated according to the parameters of the current deep Q-network to be trained.

[0064] The preset number of rounds can be set according to the actual situation; for example, the preset number of rounds is 1000.

[0065] As an example, during the training of the deep Q-network to be trained, in response to determining that after a preset number of iterations, the parameters of the current deep Q-network to be trained are copied to the target deep Q-network to synchronize the parameters in the target deep Q-network with the parameters in the deep Q-network to be trained.

[0066] Second, the target Q-value is determined by using the updated target depth Q-network based on training samples randomly selected from the training sample set.

[0067] In the DDQN (Double Deep Q-Network) algorithm, the operation of copying the weights of the deep Q-network to be trained into the target deep Q-network can solve the overestimate problem in the DQN algorithm and help improve the accuracy of the target Q-value calculation.

[0068] In traditional Q-learning, the Q-value of the current state is updated by maximizing the maximum Q-value of the next state. However, in practical applications, due to various factors, the Q-value is often overestimated, leading the agent to take incorrect actions and exhibiting instability and oscillations during the learning process.

[0069] The key idea behind the DDQN algorithm is to use a target network (a target depth Q-network) to compute the maximum Q-value for the next state, thereby reducing overestimation of the Q-value. Specifically, at regular intervals or preset rounds, the weights of the current training depth Q-network are copied to the target depth Q-network. Then, in the next interval or preset rounds, the target depth Q-network is used to compute the maximum Q-value for the next state, instead of the current network. This introduces a difference between the training depth Q-network and the target depth Q-network, thus reducing overestimation of the Q-value and improving the agent's performance and stability.

[0070] It's important to note that the update frequency of the target depth Q-network should not be too high, otherwise it will interfere with the current network's optimization process, leading to a decline in learning performance. At the same time, the target depth Q-network also needs time to adapt to changes in the depth Q-network being trained, so the update frequency should not be too low either.

[0071] Continue to refer to Figure 4 The illustration shows a schematic flow 400 of a method for determining first state information according to this embodiment, including the following steps:

[0072] Step 401: Extract the target information from the code recommendation request.

[0073] In this embodiment, the aforementioned execution entity can receive code recommendation requests and extract target information from the code recommendation requests. The target information includes: the code development language, the length of the code development documentation, the user's reaction to previously recommended code, the duration between two consecutive identical user reactions, the presence of a space before the cursor, the end position of the command prompt, the length of the last line of code corresponding to the command prompt, and the length of the last line of code corresponding to the command prompt after removing spaces.

[0074] As an example, the code development language is represented by s0; the length of the code development documentation is represented by s1; the user's reaction to the previously recommended code (accept or reject) is represented by s2; the duration (in milliseconds) between two consecutive identical user reactions (two acceptances or two rejections) is represented by s3; whether there is a space before the cursor is represented by s4; the end position of the command prompt is represented by s5; the length of the last line of code in the command prompt is represented by s6; and the length of the last line of code corresponding to the command prompt, excluding spaces, is represented by s7.

[0075] In some optional implementations of this embodiment, the code development language is represented by the weight corresponding to the code development language in a preset language weight list.

[0076] As an example, the preset language weight list includes various types of code development languages ​​and their respective weights. The weight of each code development language may be related to factors such as its versatility and the proportion of code it represents in the user's development code.

[0077] In this implementation, a pre-set list of language weights is used to represent the code development language, which helps to improve the accuracy of the represented state information and the convenience of calculation during the state transition process.

[0078] Step 402: Construct a state matrix based on the target information, which serves as the first state information.

[0079] As an example, a state matrix S is constructed based on the target information. t = [s0, s1, s2, s3, s4, s5, s6, s7], as the first state information.

[0080] In some examples, the above-mentioned execution entity can normalize s1, s3, s5, s6, and s7 to the interval [0, 1]; s2 is assigned a value based on whether it is adopted, and s4 is assigned a value based on whether it exists, specifically 0 or 1.

[0081] This embodiment provides a specific implementation method for determining the first state information, which can be applied to determining the first state information in the above embodiment 200. Through each target information, the user's code development environment state can be accurately characterized, which helps to improve the accuracy of the training process of the request filtering model.

[0082] Continue to refer to Figure 5 The illustration shows a schematic flow 500 of a method for determining reward information according to this embodiment, including the following steps:

[0083] Step 501: Determine the user's reaction to the recommended code generated by the code recommendation model and the reaction time of the user's reaction.

[0084] As an example, the user's reaction action of whether or not to adopt the recommended code is represented by R. a R a =0 indicates rejection, R s =1 indicates acceptance; the time from when a user sees the recommendation code to when they choose to accept or reject it is represented by R. b Among them, the duration R b The unit is milliseconds.

[0085] Step 502: Determine reward information based on the reaction action and reaction time using a preset reward function.

[0086] In this embodiment, the aforementioned execution entity can use the following reward function to calculate the reward information:

[0087]

[0088] In this embodiment, by adapting the reward function to the code recommendation scenario, the reward information is determined based on the user's reaction action and reaction time to the recommended code, which improves the accuracy of the reward information calculation and helps to improve the accuracy of the request filtering model obtained by the training method shown in Embodiment 200.

[0089] Continue to refer to Figure 6 The information processing flow 600 of the training method for the request filtering model is shown.

[0090] The information processing flow is as follows:

[0091] 1. During the user's code development process using the IDE, the IDE generates a code recommendation request. The code recommendation request includes the following target information: the code development language, the length of the code development documentation, the user's reaction to the previously recommended code, the duration between two consecutive identical user reactions, whether there is a space before the cursor, the end position of the command prompt, the length of the last line of code corresponding to the command prompt, and the length of the last line of code corresponding to the command prompt after removing spaces.

[0092] 2. Based on the code recommendation request, generate the first state information S. t The state matrix.

[0093] 3. Input the state matrix into the deep Q-network to be trained to determine the actual Q-value of each response action (filtering code recommendation requests or receiving code recommendation requests); and based on the actual Q-value, use a greedy strategy to determine the response action a to be executed. t Determine the reward information R based on the response action. t The second state information S updated from the first state information t+1 ; Construct training samples (S) t a t R t S t+1 The constructed training samples are then stored in the sample space.

[0094] Repeat steps 1-3 above iteratively to obtain the training sample set.

[0095] 4. Randomly select training samples from the training sample set, and calculate the target Q-value using the target depth Q-network based on the selected training samples using the following formula:

[0096] Target Q = r + γmax(Q(s) t+1 a t+1 ))

[0097] Where Target Q represents the target Q value; r represents the reward information obtained after performing the response action a; γ is a discount factor between 0 and 1, used to balance the importance of current rewards and future rewards; max(Q(s) t+1 a t+1 )) indicates the second state information s t+1 Choose the action a that maximizes the Q value from all available actions. t+1 The corresponding Q value.

[0098] 5. Calculate the loss between the actual Q-value and the target Q-value corresponding to the selected training samples, and update the parameters of the deep Q-network to be trained based on the loss.

[0099] 6. Iteratively execute steps 4-5 above. In response to each preset time length or preset iteration round, copy the parameters of the current training depth Q network to the target depth Q network and update the parameters of the target depth Q network.

[0100] Upon reaching a preset termination condition, the trained Q-network of untrained depth is designated as the request filtering model. The preset termination condition could be, for example, training time exceeding a preset time threshold, the number of training iterations exceeding a preset number of iterations threshold, or the training loss converging.

[0101] Continue to refer to Figure 7 The illustration shows a schematic flow 700 of an embodiment of the request filtering method according to the present disclosure, including the following steps:

[0102] Step 701: Obtain the code recommendation request.

[0103] In this embodiment, the execution body of the request filtering method (e.g., Figure 1 The terminal device or server in the system can obtain code recommendation requests remotely or locally via wired network connection or wireless network connection.

[0104] The code recommendation request includes the following target information: the code development language, the length of the code development documentation, the user's reaction to the previously recommended code, the duration between two consecutive identical user reactions, whether there is a space before the cursor, the end position of the command prompt, the length of the last line of code corresponding to the command prompt, and the length of the last line of code corresponding to the command prompt after removing spaces.

[0105] Step 702: Using the trained request filtering model, determine the response action that represents whether to filter code recommendation requests.

[0106] In this embodiment, the aforementioned execution entity can determine the response action representing whether to filter code recommendation requests using the trained request filtering model. The request filtering model is trained using the model described in embodiment 200.

[0107] As an example, the aforementioned executing entity can construct a state matrix [s0, s1, s2, s3, s4, s5, s6, s7] based on the following: code development language (s0), code development document length (s1), user's reaction action to the previous recommended code (accept or reject) (s2), duration (in milliseconds) between two consecutive identical user reactions (two acceptances or two rejections) (s3), presence of a space before the cursor (s4), end position of the command prompt (s5), length of the last line of code in the command prompt (s6), and length of the last line of code corresponding to the command prompt (excluding spaces) (s7). This state matrix is ​​then input into the request filtering model to obtain the response action representing whether to filter the code recommendation request.

[0108] The response actions specifically include filtering code recommendation requests and receiving code recommendation requests.

[0109] This embodiment provides a request filtering method. The request filtering model obtained based on deep reinforcement learning can accurately filter out inferior code recommendation requests and improve the code recommendation efficiency of the code recommendation process.

[0110] In some optional implementations of this embodiment, the execution entity may also perform the following operations: in response to determining the response action representation to receive the code recommendation request, generate recommended code through the code recommendation model.

[0111] Common code recommendation models include rule-based, model-based, and deep learning-based models. Rule-based models are the simplest, recommending common code snippets such as constants, frequently used functions, and common variables. Model-based models build a model based on the user's programming habits and historical data, then recommend relevant code snippets based on the model's predictions. Deep learning-based models utilize deep learning algorithms to learn from the user's programming habits and historical data, thus building a more accurate recommendation model.

[0112] In this implementation, code recommendation is only performed in response to a determined response action characterization of receiving a code recommendation request, thus improving the effectiveness of the code recommendation process.

[0113] Continue to refer to Figure 8 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a training apparatus for a request filtering model, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0114] like Figure 8As shown, the training device 800 for the request filtering model includes: an accumulation unit 801, configured to accumulate experience on whether to filter code recommendation requests through a deep Q-network to be trained using a greedy strategy during the code recommendation process, generating a training sample set, wherein each training sample in the training sample set includes first state information, a response action representing whether to filter code recommendation requests, reward information based on the response action, and second state information, wherein the first state information represents the environmental state of the code development process, the second state information represents the first state information after being changed based on the response action, the response action is determined by the actual Q-value output by the deep Q-network to be trained according to the first state information, and the actual Q-value represents the cumulative expected reward for the response action under the first state information; a first determination unit 802, configured to determine a target Q-value through a target deep Q-network based on training samples randomly selected from the training sample set, wherein the target Q-value represents the maximum expected cumulative reward for the response action in the selected training samples; and an update unit 803, configured to update the deep Q-network to be trained according to the loss between the actual Q-value and the target Q-value corresponding to the same training sample in the training sample set, so as to finally obtain the request filtering model.

[0115] In some optional implementations of this embodiment, the accumulation unit 801 is further configured to: determine the first state information by: extracting target information from the code recommendation request, wherein the target information includes: the code development language, the length of the code development document, the user's reaction to the previously recommended code, the duration between two consecutive identical user reactions, whether there is a space before the cursor, the end position of the command prompt, the length of the last line of code corresponding to the command prompt, and the length of the last line of code corresponding to the command prompt after removing spaces; and constructing a state matrix based on the target information as the first state information.

[0116] In some optional implementations of this embodiment, the code development language is represented by the weight corresponding to the code development language in a preset language weight list.

[0117] In some optional implementations of this embodiment, the first determining unit 802 is further configured to: during the training process of the depth Q network to be trained, in response to determining that after each preset number of iterations, update the target depth Q network according to the parameters of the current depth Q network to be trained; and determine the target Q value based on training samples randomly selected from the training sample set using the updated target depth Q network.

[0118] In some optional implementations of this embodiment, the accumulation unit 801 is further configured to: determine reward information by: determining the user's reaction action to the recommended code generated by the code recommendation model and the reaction time of the user's reaction action; and determining reward information based on the reaction action and reaction time using a preset reward function.

[0119] This embodiment provides a training device for a request filtering model. It employs a DDQN (Double Deep Q-network) architecture based on a training deep Q-network and a target deep Q-network. During the code recommendation process, the training deep Q-network uses a greedy strategy to accumulate experience regarding whether to filter code recommendation requests. The training deep Q-network is updated based on the loss between the actual Q-value obtained from the training deep Q-network and the target Q-value obtained from the target deep Q-network. This results in a request filtering model with the function of filtering code recommendation requests, which can accurately filter low-quality code recommendation requests and improve the efficiency and effectiveness of the code recommendation process.

[0120] Continue to refer to Figure 9 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a request filtering method, which is similar to... Figure 7 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0121] like Figure 9 As shown, the request filtering device 900 includes: an acquisition unit 901 configured to acquire code recommendation requests; and a second determination unit 902 configured to determine a response action characterizing whether to filter the code recommendation request using a trained request filtering model, wherein the request filtering model is trained using the above embodiment 800.

[0122] In some optional implementations of this embodiment, the apparatus further includes: a recommendation unit (not shown in the figure), configured to receive a code recommendation request in response to determining a response action characterization, and generate recommended code through a code recommendation model.

[0123] In this embodiment, a request filtering device is provided. Based on a request filtering model obtained by a deep reinforcement learning method, it can accurately filter out inferior code recommendation requests and improve the code recommendation efficiency of the code recommendation process.

[0124] According to embodiments of this disclosure, this disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to implement the training method and request filtering method of the request filtering model described in any of the above embodiments.

[0125] According to embodiments of this disclosure, this disclosure also provides a readable storage medium storing computer instructions that enable a computer to implement the training method and request filtering method of the request filtering model described in any of the above embodiments when executed.

[0126] This disclosure provides a computer program product that, when executed by a processor, can implement the training method and request filtering method of the request filtering model described in any of the above embodiments.

[0127] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0128] like Figure 10 As shown, device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1002 or a computer program loaded from storage unit 1008 into random access memory (RAM) 1003. The RAM 1003 may also store various programs and data required for the operation of device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0129] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0130] The computing unit 1001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as the request filtering model training method. For example, in some embodiments, the request filtering model training method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the request filtering model training method described above can be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured in any other suitable manner (e.g., by means of firmware) to perform a training method for the request filtering model.

[0131] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0132] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0133] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0134] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0135] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0136] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, also known as cloud computing servers or cloud hosts, which are hosting products within the cloud computing service system to address the management difficulties and weak business scalability inherent in traditional physical hosts and Virtual Private Servers (VPS) services; they can also be servers for distributed systems or servers integrated with blockchain technology.

[0137] According to the technical solution of this disclosure, a training method for a request filtering model is provided. This method employs a DDQN (Double Deep Q-network) architecture based on a training deep Q-network and a target deep Q-network. During the code recommendation process, the training deep Q-network uses a greedy strategy to accumulate experience regarding whether to filter code recommendation requests. The training deep Q-network is updated based on the loss between the actual Q-value obtained from the training deep Q-network and the target Q-value obtained from the target deep Q-network. This results in a request filtering model with the function of filtering code recommendation requests, which can accurately filter inferior code recommendation requests and improve the efficiency of the code recommendation process.

[0138] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution provided in this disclosure can be achieved, and this is not limited herein.

[0139] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for training a request filtering model, comprising: In the code recommendation process, a training sample set is generated by accumulating experience on whether to filter code recommendation requests through a greedy strategy using a deep Q-network to be trained. Each training sample in the training sample set includes first state information, a response action representing whether to filter the code recommendation request, reward information based on the response action, and second state information. The first state information represents the environmental state of the code development process, and the second state information represents the first state information after it has changed based on the response action. The response action is determined by the deep Q-network to be trained based on the actual Q value output by the first state information. The actual Q value represents the cumulative expected reward for the response action under the first state information. By using a target depth Q-network, a target Q-value is determined based on training samples randomly selected from the training sample set, wherein the target Q-value represents the maximum expected cumulative reward of the response action in the selected training samples; Based on the loss between the actual Q value and the target Q value corresponding to the same training sample in the training sample set, the deep Q network to be trained is updated to finally obtain the request filtering model; The first status information is determined in the following way: Extract target information from the code recommendation request, wherein the target information includes: code development language, length of code development documentation, user's reaction to previously recommended code, duration between two consecutive identical user reactions, presence of a space before the cursor, end position of the command prompt, length of the last line of code corresponding to the command prompt, and length of the last line of code corresponding to the command prompt after removing spaces. Based on the target information, a state matrix is ​​constructed as the first state information; The reward information is determined in the following manner: Determine the user's reaction to the recommended code generated by the code recommendation model and the duration of the user's reaction to the action. The reward information is determined based on the reaction action and the reaction time using a preset reward function.

2. The method according to claim 1, wherein, The code development language is represented by the weight corresponding to the code development language in a preset language weight list.

3. The method according to claim 1, wherein, The step of determining the target Q-value using a target depth Q-network based on training samples randomly selected from the training sample set includes: During the training process of the deep Q-network to be trained, in response to determining that after each preset number of iterations, the target deep Q-network is updated according to the current parameters of the deep Q-network to be trained; The target Q-value is determined using the updated target depth Q-network based on training samples randomly selected from the training sample set.

4. A request filtering method, comprising: Get code recommendation request; The trained request filtering model determines whether to filter the code recommendation request response action, wherein the request filtering model is trained using any one of claims 1-3.

5. The method according to claim 4, wherein, Also includes: In response to determining that the response action characterizes receiving the code recommendation request, recommended code is generated through the code recommendation model.

6. A training apparatus for a request filtering model, comprising: An accumulation unit is configured to accumulate experience regarding whether to filter code recommendation requests through a greedy strategy during the code recommendation process using a deep Q-network to be trained, generating a training sample set. Each training sample in the training sample set includes first state information, a response action representing whether to filter the code recommendation request, reward information based on the response action, and second state information. The first state information represents the environmental state of the code development process, and the second state information represents the first state information after being changed based on the response action. The response action is determined by the actual Q-value output by the deep Q-network to be trained based on the first state information, and the actual Q-value represents the cumulative expected reward for the response action under the first state information. The first determining unit is configured to determine a target Q value through a target depth Q network based on training samples randomly selected from the training sample set, wherein the target Q value represents the maximum expected cumulative reward of the response action in the selected training samples. The update unit is configured to update the deep Q-network to be trained based on the loss between the actual Q-value and the target Q-value corresponding to the same training sample in the training sample set, so as to finally obtain the request filtering model. The accumulation unit is further configured to determine the first state information in the following manner: Extract target information from the code recommendation request, wherein the target information includes: the code development language, the length of the code development documentation, the user's reaction to the previously recommended code, the duration between two consecutive identical user reactions, whether there is a space before the cursor, the end position of the command prompt, the length of the last line of code corresponding to the command prompt, and the length of the last line of code corresponding to the command prompt after removing spaces; construct a state matrix based on the target information, as the first state information; The accumulation unit is further configured to determine the reward information in the following manner: Determine the user's reaction to the recommended code generated by the code recommendation model and the duration of the user's reaction; determine the reward information based on the reaction and the reaction duration using a preset reward function.

7. The apparatus according to claim 6, wherein, The code development language is represented by the weight corresponding to the code development language in a preset language weight list.

8. The apparatus according to claim 6, wherein, The first determining unit is further configured to: During the training process of the deep Q-network to be trained, in response to determining that after each preset number of iterations, the target deep Q-network is updated according to the current parameters of the deep Q-network to be trained; The target Q-value is determined using the updated target depth Q-network based on training samples randomly selected from the training sample set.

9. A request filtering device, comprising: The retrieval unit is configured to retrieve code recommendation requests; The second determining unit is configured to determine, through a trained request filtering model, a response action characterizing whether to filter the code recommendation request, wherein the request filtering model is trained by any one of claims 6-8.

10. The apparatus according to claim 9, wherein, Also includes: The recommendation unit is configured to receive the code recommendation request in response to determining the response action characterization, and generate recommended code through the code recommendation model.

11. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

12. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-5.

13. A computer program product, comprising: A computer program that, when executed by a processor, implements the method of any one of claims 1-5.

Images