A type inference method and apparatus based on reinforcement learning
Through the type inference method based on reinforcement learning, the generation model is fine-tuned by static feedback, the problem of type inference model prediction bias in dynamic type programming language is solved, and the accuracy and security of type prediction are improved.
Patent Information
- Application Number
- CN202410668413.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-28
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-05-28
AI Technical Summary
When the existing type inference model deals with dynamic type programming languages, especially for variables and dynamic evaluation functions with fewer static constraints, the prediction results are biased greatly from actual applications, and cannot effectively ensure type safety and compilation performance.
Using a type inference method based on reinforcement learning, the operation environment data and code data are obtained for preprocessing, legal type sequences and type flow sequences are generated, and supervision training is carried out for code generation tasks with type annotations, and reinforcement learning is trained using static feedback. , fine-tune the generative model to avoid type errors.
It improves the accuracy of type prediction, enhances the type prediction ability of the generative model, solves the problem that the prediction results of the existing type inference model are biased from the actual application, and ensures type safety and compilation performance.
Smart Images

Figure CN118246556B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of reinforcement learning, and in particular to a type inference method and device based on reinforcement learning. Background Art
[0002] In recent years, dynamically typed programming languages like Python have become increasingly popular. The dynamic nature gives greater flexibility to programming and makes rapid prototyping possible. However, dynamically typed programming languages face some challenges in ensuring safety and compilation performance due to the lack of well-defined variable types. According to a recent survey, one of the most desired features for Python developers is static typing or at least some strict type hints. Type prediction is a common task. Traditional static type inference methods and type inference tools (such as the Pytype tool, the Pysonar2 tool, and the Pyre Infer tool) can correctly infer the types of variables with sufficient static constraints. For example, for "a = 1", it can be determined that the type of "a" is int (an identifier used to define an integer type variable), but they cannot handle variables with fewer static constraints, such as most function parameters or dynamic evaluations such as the eval() function.
[0003] Traditional type inference methods use a rule-based approach to accurately predict the type of each type slot. Some studies have introduced probabilistic type inference, returning multiple candidate types. Other models treat types as word labels and use sequence models for inference, but there is a limitation that each variable occurrence is considered a new variable. On the other hand, some studies have introduced conceptual types, further subdividing a single type, and designed multiple sequence models to infer function types with a validation phase to filter out incorrect predictions. Another method uses a graphical model to represent the code and uses KNN (nearest neighbor algorithm) to predict types, but it fails when the predicted type is not in the training set.
[0004] Therefore, the prior art needs to be improved. Summary of the invention
[0005] The technical problem to be solved by the present invention is that, in view of the defects of the prior art, the present invention provides a type inference method and device based on reinforcement learning to solve the technical problem that the prediction results of the type inference model deviate greatly from the actual application.
[0006] The technical solution adopted by the present invention to solve the technical problem is as follows:
[0007] In a first aspect, the present invention provides a type inference method based on reinforcement learning, comprising:
[0008] Acquire operating environment data and code data, and preprocess the operating environment data and the code data to obtain a legal type sequence and a type stream sequence;
[0009] Performing supervised training of a code generation task with type annotations according to the legal type sequence and the type flow sequence;
[0010] According to the generated target code with type annotations, static feedback reinforcement learning training is performed to obtain a reinforced code generation model.
[0011] In one implementation, the preprocessing of the operating environment data and the code data includes:
[0012] Perform runtime environment dependency inference to obtain the corresponding third-party package versions;
[0013] Based on the defined user types and / or the user types imported by the third-party package version, analyze to obtain all user-defined types and obtain the legal type sequence;
[0014] Data flow analysis is performed on the abstract syntax tree of the target function to obtain all data flows, and all data flows are screened to obtain the type flow sequence.
[0015] In one implementation, the performing supervised training of the code generation task with type annotations according to the legal type sequence and the type flow sequence includes:
[0016] Generate a training data set according to the legal type sequence, the type flow sequence, and a public code example with type annotations;
[0017] The encoder and decoder are supervisedly trained for a code generation task with type annotations according to the training data set to obtain a supervised model.
[0018] In one implementation, the static feedback reinforcement learning training is performed according to the generated target code with type annotation to obtain a reinforced code generation model, which includes:
[0019] Build a reinforcement learning environment based on the labeled data for reinforcement learning training;
[0020] A reinforcement learning model is constructed, and an end-of-round reward and a relative entropy reward are set based on the reinforcement learning model.
[0021] In one implementation, the static feedback reinforcement learning training is performed according to the generated target code with type annotation to obtain a reinforced code generation model, including:
[0022] Aggregate samples that are predicted incorrectly by the supervised model into a difficult sample dataset;
[0023] Sampling common samples of the same size as the difficult samples from all samples, and combining the difficult sample data set and the common samples to form a mixed data set;
[0024] Initialize the model using the weights of the supervised model, and train the supervised model on the difficult sample data set using a reinforcement learning algorithm to obtain a first model;
[0025] Initializing the model with the weights of the first model, and training the model on the mixed data set using reinforcement learning with supervised loss to obtain a second model;
[0026] The supervised model is used as a student model, and the second model is used as a teacher model for training to obtain the enhanced code generation model.
[0027] In one implementation, the supervised loss of the reinforcement learning with supervised loss is: calculating the cross entropy loss between the training model output and the sample label, and adding the calculated cross entropy loss to the reinforcement learning loss in a certain ratio to obtain the total loss.
[0028] In one implementation, the reinforcement learning algorithm is a proximal strategy optimization algorithm, the reinforcement learning algorithm uses an actor-critic architecture, and the network model of the reinforcement learning algorithm includes: a strategy output layer and a value output layer; the strategy output layer is used to output the probability distribution of executable actions of the agent in the current state, and the value output layer is used to output the value of the current state.
[0029] In a second aspect, the present invention provides a type inference device based on reinforcement learning, comprising:
[0030] A preprocessing module, used for acquiring operating environment data and code data, and preprocessing the operating environment data and the code data to obtain a legal type sequence and a type flow sequence;
[0031] A supervised training module, used for performing supervised training of a code generation task with type annotations according to the legal type sequence and the type flow sequence;
[0032] The reinforcement learning module is used to perform static feedback reinforcement learning training based on the generated target code with type annotations to obtain a reinforced code generation model.
[0033] In a third aspect, the present invention provides a computer terminal, comprising: a processor and a memory, wherein the memory stores a type inference program based on reinforcement learning, and when the type inference program based on reinforcement learning is executed by the processor, it is used to implement the operation of the type inference method based on reinforcement learning as described in the first aspect.
[0034] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a reinforcement learning-based type inference program, and when the reinforcement learning-based type inference program is executed by a processor, it is used to implement the operation of the reinforcement learning-based type inference method as described in the first aspect.
[0035] The present invention adopts the above technical solution to achieve the following effects:
[0036] The present invention uses reinforcement learning with feedback from a static type checker to fine-tune the generative model. In the model trained by supervised learning, type predictions of variables in the software can be obtained and passed to the static type checker to see if there are any type conflicts. Then, reinforcement learning with a specific reward function and training algorithm is used to fine-tune the generative model to avoid knowledge of type errors; the present invention proposes to use a generation-based method to predict types, and enhance the accuracy of type prediction of the generative model through reinforcement learning, solving the problem that the prediction results of the existing type inference model deviate greatly from the actual application. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.
[0038] Figure 1 It is a flowchart of a type inference method based on reinforcement learning in one implementation of the present invention.
[0039] Figure 2 It is a model architecture diagram in one implementation of the present invention.
[0040] Figure 3 It is a diagram of an encoder-decoder architecture based on T5 in one implementation of the present invention.
[0041] Figure 4 It is a schematic diagram of the reinforcement learning process in one implementation of the present invention.
[0042] Figure 5It is a schematic diagram of a reinforcement learning model structure in one implementation of the present invention.
[0043] Figure 6 It is a schematic diagram of the knowledge distillation process in one implementation of the present invention.
[0044] Figure 7 It is a functional principle diagram of a terminal in one implementation of the present invention.
[0045] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0046] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0047] Exemplary Methods
[0048] Traditional type inference methods use a rule-based approach to accurately predict the type of each type slot. Some studies have introduced probabilistic type inference, returning multiple candidate types. Other models treat types as word labels and use sequence models for inference, but there is a limitation that each variable occurrence is considered a new variable. On the other hand, some studies have introduced conceptual types, further subdividing a single type, and designed multiple sequence models to infer function types with a validation phase to filter out incorrect predictions. Another method uses a graphical model to represent the code and uses KNN (nearest neighbor algorithm) to predict types, but it fails when the predicted type is not in the training set.
[0049] In response to the above technical problems, an embodiment of the present invention provides a type inference method based on reinforcement learning, which uses reinforcement learning with feedback from a static type checker to fine-tune the generation model. In a model trained by supervised learning, type predictions of variables in the software can be obtained and passed to a static type checker to see if there are any type conflicts. Then, reinforcement learning with a specific reward function and training algorithm is used to fine-tune the generation model to avoid knowledge of type errors; therefore, an embodiment of the present invention proposes using a generation-based method to predict types, and enhancing the accuracy of type prediction of the generation model through reinforcement learning, which solves the problem that the prediction results of the existing type inference model deviate greatly from actual applications.
[0050] like Figure 1 As shown, an embodiment of the present invention provides a type inference method based on reinforcement learning, comprising the following steps:
[0051] Step S100, acquiring operating environment data and code data, and preprocessing the operating environment data and the code data to obtain a legal type sequence and a type flow sequence.
[0052] This embodiment discloses a type inference method based on reinforcement learning, which uses a generation-based method to predict types and enhances the accuracy of type prediction of the generation model through reinforcement learning. The method includes two main stages: generation-based type prediction and reinforcement learning with static feedback.
[0053] First, this embodiment uses a generation-based approach to solve the problem of low-frequency type prediction. However, the generation-based approach introduces two problems: 1) Since the types are generated from a vocabulary rather than a fixed type set, the generated types may be invalid; 2) Since the generation model generates types for all variables in a function as a type sequence, type conflicts may occur.
[0054] To address these accuracy issues, this embodiment proposes to use reinforcement learning with feedback from a static type checker to fine-tune the generative model. In a model trained with supervised learning, type predictions for variables in the software can be obtained and passed to a static type checker to see if there are type conflicts. Then, reinforcement learning with a specific reward function and training algorithm is used to fine-tune the generative model to avoid knowledge of type errors.
[0055] Specifically, in one implementation of this embodiment, step S100 includes the following steps:
[0056] Step S101, performing runtime environment dependency inference to obtain a corresponding third-party package version;
[0057] Step S102, based on the defined user types and / or the user types imported by the third-party package version, analyzing to obtain all user-defined types, and obtaining the legal type sequence;
[0058] Step S103, performing data flow analysis on the abstract syntax tree of the target function to obtain all data flows, screening all data flows, and obtaining the type flow sequence.
[0059] In this embodiment, before the generated type prediction is performed, the operating environment data and the code data need to be preprocessed. The preprocessing process may include the following two aspects:
[0060] 1. Import analysis
[0061] Since built-in types are a set of fixed types predefined in the language specification, and user-defined types are defined by developers in the current software or imported from third-party packages. Therefore, in this embodiment, it is necessary to import user-defined types in third-party packages and perform analysis to obtain all user-defined types. If the imported module comes from a third-party package, the runtime environment dependency inference is first performed to obtain the correct third-party package version and download it.
[0062] By obtaining the types defined in the current software or importing user-defined types in third-party packages, a legal type sequence for supervised training of generative models is obtained.
[0063] 2. Type flow analysis
[0064] This embodiment represents the context of a function through a series of type flows. To obtain the type flow, first perform traditional data flow analysis on the abstract syntax tree of the target function to obtain all data flows, starting from the function parameters to the function return value. Then, all data flows are screened, and unnecessary branches are pruned and merged, that is, branches with different values of the indicator variables but the same type; finally, a type flow sequence for supervised training of the generative model is obtained.
[0065] like Figure 1 As shown, in one implementation of the embodiment of the present invention, the type inference method based on reinforcement learning also includes the following steps:
[0066] Step S200: performing supervised training of a code generation task with type annotations based on the legal type sequence and the type flow sequence.
[0067] This embodiment proposes a sequence model based on an encoder-decoder architecture, such as Figure 2 As shown in the figure, it is used to perform the code generation task with type annotation. The model accepts the legal type sequence parsed by the runtime environment and the abstract syntax tree of the target code as input, and generates the target code with type annotation. Through the supervised training method, the model is trained on a public code dataset with type annotation, achieving high-quality code generation.
[0068] Specifically, in one implementation of this embodiment, step S200 includes the following steps:
[0069] Step S201, generating a training data set according to the legal type sequence, the type flow sequence and the public code examples with type annotations;
[0070] Step S202: Perform supervised training of a code generation task with type annotations on the encoder and decoder according to the training data set to obtain a supervised model.
[0071] In this embodiment, the training data set has data from three sources, including legal type sequences parsed by the runtime environment, abstract syntax trees parsed from the target code, and publicly available code examples with type annotations. During the data set preparation process, the data from the above three sources are integrated into a pair of input-output samples. Each sample consists of an input sequence and an output sequence. The input sequence is represented by a legal type sequence and an abstract syntax tree, and the output sequence is a target code with type annotations. Therefore, the training data set can be represented as:
[0072] ;
[0073] in, represents the number of training samples, It is The input sequence of samples is is the corresponding output sequence.
[0074] This embodiment adopts an encoder-decoder architecture based on T5 (Text-to-Text Transfer Transformer, a model architecture), such as Figure 3 As shown, it is used to transform the input sequence into a representation in the semantic space and generate an output sequence.
[0075] The encoder is composed of 12 identical layers stacked together. Each layer contains two sub-layers. The first sub-layer is a multi-head self-attention mechanism, and the second sub-layer is a fully connected feed-forward neural network. In each layer, the word embedding representation of the input sequence , calculate the projection of query, key and value. Then, calculate the multi-head self-attention:
[0076] ;
[0077] The calculation of each attention head is as follows:
[0078] ;
[0079] in, , , are the projection matrices of queries, keys, and values, is the number of attention heads, is the output projection matrix, is the dimension of the model. Then at the output of each attention head A feedforward neural network is applied to extract more abstract, contextual information from the raw input, which helps to better understand and process sequence data to obtain the final encoder layer output:
[0080] ;
[0081] in and is the weight matrix used to map input data to hidden representations and final output. and is the bias vector, which is used to introduce an offset to adjust the output of the network. It is a rectified linear unit function that performs an element-by-element nonlinear transformation on the input, converting values less than zero to zero and keeping values greater than zero unchanged.
[0082] The decoder is a key part of generating the output sequence, which also includes multi-head self-attention and feedforward neural networks. The task of the decoder is to generate the final output sequence based on the encoded semantic representation and the previously generated partial sequence. The difference is that position masking is required in the decoder to prevent the current position from paying attention to future information. When the decoder obtains the generated result Finally, a Softmax (normalized exponential function) classifier is used to determine the probability distribution of each word in the generated sequence, thereby selecting the next most likely word. The calculation formula is:
[0083] ;
[0084] in, The first elements, and the result represents the first element obtained after Softmax calculation. The probability of an element.
[0085] In this embodiment, after supervised training based on the generative model, all samples predicted by the supervised model are classified to obtain samples with incorrect predictions and samples with correct predictions, so as to facilitate the subsequent establishment of a difficult sample data set and a mixed data set for reinforcement learning.
[0086] like Figure 1 As shown, in one implementation of the embodiment of the present invention, the type inference method based on reinforcement learning also includes the following steps:
[0087] Step S300: Perform static feedback reinforcement learning training based on the generated target code with type annotations to obtain a reinforced code generation model.
[0088] In this embodiment, reinforcement learning is a machine learning method that focuses on how an agent interacts with the environment to maximize rewards. In reinforcement learning, there are two objects that can interact: the agent and the environment. The agent can perceive the state of the environment and choose appropriate actions based on the state of the environment feedback. The environment will execute the action selected by the agent and generate a reward signal to the agent based on the result of executing the action. The agent optimizes its own strategy through the reward signal to maximize the total reward. The reinforcement learning process is as follows: Figure 4 shown.
[0089] The reinforcement learning part of this invention will be introduced from the aspects of reinforcement learning environment, reinforcement learning algorithm, reinforcement learning reward design, and reinforcement learning training process.
[0090] Specifically, in one implementation of this embodiment, the following steps are included before step S300:
[0091] Step S301a, constructing a reinforcement learning environment based on the labeled data for reinforcement learning training;
[0092] Step S301b, constructing a reinforcement learning model, and setting a round end reward and a relative entropy reward based on the reinforcement learning model.
[0093] In this embodiment, a reinforcement learning environment is constructed, and reinforcement learning requires the intelligent agent to interact with the environment for learning. The present invention constructs a reinforcement learning environment based on the labeled data for reinforcement learning training; wherein the reinforcement learning environment includes two parts: the state and action of the reinforcement learning environment and the environment operation process of the reinforcement learning environment.
[0094] 1. Reinforce the state and actions of the learning environment
[0095] The agent decides what action to perform based on the state of the environment.
[0096] 1) The state of the reinforcement learning environment consists of the following parts:
[0097] - input_ids (input): The input text is converted into a sequence of token numbers (token_id) through a tokenizer, which is used to represent the input text. Token_id is the subscript of the word in the vocabulary after tokenization;
[0098] - attention_mask: marks which input_ids are real input words and which are padding, used for the attention mechanism;
[0099] - decoder_input_ids (encoder input): the token_id sequence of the currently generated response;
[0100] - decoder_attention_mask (encoding attention mask): marks the real generated words and padding in decoder_input_ids;
[0101] - predicted_len (predicted length): the length of the currently generated response (number of tokens).
[0102] 2) The action accepted by the reinforcement learning environment is an integer representing the token_id of the next word predicted to be generated.
[0103] 2. Strengthen the environment operation process of the learning environment
[0104] 1) Episode starts: The environment reads an input data from the training set, which contains processed features and labels. After processing the data, the environment initializes the episode and generates an initial state;
[0105] 2) The model selects an action based on the state (predicting the token_id of the next word);
[0106] 3) The environment saves the token_id output by the model, generates a new state, calculates the reward, and determines whether the round is terminated;
[0107] - Round termination conditions:
[0108] 3.1) The process ends when the length of the generated prediction sequence reaches the maximum length.
[0109] 3.2) It ends when a special termination token is generated, including: sep_token (separator) of the tokenizer; eos_token (end-of-sentence token) of the tokenizer.
[0110] 3.3) End when the predicted word id is empty or invalid.
[0111] 4) Repeat 2) to 4) until the round is over;
[0112] 5) End of round: A type is restored based on the saved token_id sequence, that is, the prediction of a type is completed.
[0113] In this embodiment, the algorithm used for reinforcement learning training is the proximal policy optimization (PPO) algorithm. The PPO algorithm uses an actor-critic architecture (i.e., Actor-Critic), and its network model contains two outputs, namely, policy output and value output. After the state is input into the reinforcement learning network, the policy output gives the probability distribution of the actions that the agent may perform in the current state, and the value output gives the value of the current state to evaluate the quality of the agent's strategy. The algorithm flow of PPO includes the following steps:
[0114] The start to end of a task is called a complete round.
[0115] (1) At each moment in a round , the agent is Always get status from the environment ;
[0116] (2) Input network model, policy network output action probability distribution, agent selects an action based on probability distribution sampling , the probability of selecting this action is ; Value network output value .
[0117] (3) The agent will act Return to the environment for execution, and the environment will give a reward after execution , and gives the state of the next moment .
[0118] (4) At each moment, the decision information , , , , Stored in the memory pool for subsequent network updates.
[0119] (5) The agent repeats the process from (1) to (4) until the round end condition is reached and the round ends.
[0120] (6) After several rounds, the algorithm calculates loss functions such as strategy loss and value loss based on the collected decision information, and updates the network model parameters based on the loss function. After several network updates, if the loss function and model strategy tend to be stable, the training is completed.
[0121] The strategy loss of the PPO algorithm is:
[0122] ;
[0123] in, , , is the advantage function:
[0124] ;
[0125] The value loss function is:
[0126] ;
[0127] The PPO algorithm learns the policy network and the value network by optimizing these two loss functions simultaneously.
[0128] In this embodiment, after building the reinforcement learning environment, it is also necessary to build a reinforcement learning model. The reinforcement learning model structure used is as follows: Figure 5 As shown in the figure, the reinforcement learning model consists of two models, namely the Actor model and the Critic model.
[0129] The Actor model consists of a T5 model and a linear layer.
[0130] 1) Load the model parameters after supervised learning before T5 model training.
[0131] 2) The linear layer contains a fully connected layer, which maps the output of the T5 model (768 dimensions) to the output space, and outputs a probability vector of the action space dimension (32100 dimensions), which represents the execution probability of all actions. The model fits a probability distribution in the action space based on these probabilities, and samples the actions to be executed on this distribution.
[0132] The Critic model consists of a T5 model and a linear layer.
[0133] 1) The model parameters after supervised learning are loaded before T5 model training, which are the same as the T5 model parameters in the Actor model.
[0134] 2) The linear layer contains a fully connected layer that maps the output of the T5 model (768 dimensions) to the output space with an output size of 1 dimension.
[0135] In this embodiment, after the reinforcement learning model is constructed, the reinforcement learning reward is set. The reinforcement learning reward is divided into two parts: the round end reward and the KL reward (i.e., the relative entropy reward). The round end reward is calculated at the end of a round, and the KL reward is calculated at each step in a round.
[0136] 1. Prediction length penalty:
[0137] ;
[0138] in, To predict the longer of the sequence length and the label length, For the shorter one.
[0139] 2. Similarity Rewards:
[0140] The similarity reward is calculated based on the similarity between the predicted type and the label type. The calculation process is:
[0141] (1) Split the prediction type and label type into a list of items;
[0142] (2) Calculate the total number of layers of the tag type. The total number of layers is determined by the " <eol>“Quantity determines;
[0143] (3) For each layer, start comparing the two lists from the first item. If different items are found, check whether the two items are equivalent. If they are not equivalent, terminate the comparison and the layers are different.
[0144] (4) If every item in a layer is the same or equivalent, then the layers are the same;
[0145] (5) Similarity Reward It is the same number of layers divided by the total number of layers, and the value is between [0, 1].
[0146] The similarity reward is further calculated based on the prediction length penalty, including:
[0147] (1) When the similarity is equal to 0, ;
[0148] (2) When the similarity reward is greater than 0, .
[0149] 3. Progress Rewards:
[0150] Calculate the similarity award of the supervised model prediction results separately and similarity rewards for training model prediction results The progress value is defined as:
[0151] .
[0152] If the progress value is greater than 0, the progress reward is calculated as:
[0153] .
[0154] Where max is The maximum value is fixed at 2.
[0155] If the progress value is less than or equal to 0, the progress reward is 0.
[0156] 4. Demotion Penalty:
[0157] If the progress value is less than 0, the regression penalty is calculated as:
[0158] .
[0159] Where max is The maximum value is fixed at 2.
[0160] If the progress value is greater than or equal to 0, the regression penalty is 0.
[0161] 5. Correct Rewards:
[0162] Correct rewards are calculated based on the following situations:
[0163] (1) If the prediction is correct but the supervised model is not, an additional reward is given, where 1 is Maximum value.
[0164] .
[0165] (2) If the prediction is wrong but the supervised model can predict it correctly, an additional penalty is given for the mistake.
[0166] .
[0167] (3) If both the supervision model and the training model make wrong predictions, a penalty is imposed, where 1 is Maximum value.
[0168] .
[0169] 6. End of Round Rewards:
[0170] At the end of a round, a type of prediction is completed, and the reward is calculated as the sum of the progress reward, regression penalty, and correct reward.
[0171] .
[0172] 7. KL Rewards:
[0173] The end-of-episode reward is calculated only once at the end of the episode, while the KL reward is calculated once for each step in an episode. The KL reward is calculated based on the KL divergence between the output distributions of the currently trained model and the initial supervised model for the same input. The KL divergence is denoted as ,in represents the output distribution of the current model, represents the output distribution of the initial supervised model. The KL divergence is specifically the difference between the logarithmic probability values of the probability distributions output by the two models on the same sample.
[0174] KL rewards are:
[0175] ;
[0176] in is the coefficient, set to 0.01.
[0177] In this embodiment, after constructing a reinforcement learning environment, selecting a reinforcement learning algorithm, building a reinforcement learning model, and setting reinforcement learning rewards, reinforcement learning training with static feedback is performed according to these configurations.
[0178] Specifically, in one implementation of this embodiment, step S300 includes the following steps:
[0179] Step S301, aggregating samples predicted incorrectly by the supervision model into a difficult sample data set;
[0180] Step S302, sampling common samples of the same size as the difficult samples from all samples, and combining the difficult sample data set and the common samples to form a mixed data set;
[0181] Step S303, using the weights of the supervised model to initialize the model, and using a reinforcement learning algorithm to train the supervised model on the difficult sample data set to obtain a first model;
[0182] Step S304, initializing the model with the weights of the first model, and training the model on the mixed data set using reinforcement learning with supervised loss to obtain a second model;
[0183] Step S305, using the supervision model as a student model and the second model as a teacher model for training to obtain the enhanced code generation model.
[0184] In this embodiment, the training process of reinforcement learning with static feedback is:
[0185] 1) Construct difficult sample data sets and mixed sample data sets:
[0186] Aggregate samples that are predicted incorrectly by the supervised model into a difficult sample dataset;
[0187] Ordinary samples of the same size as the difficult samples are sampled from all samples to form a mixed data set.
[0188] 2) Reinforcement Learning:
[0189] Initialize the model using the supervised model weights, and train the model using reinforcement learning on the difficult sample dataset to obtain Model 1 (i.e., the first model). The hyperparameter settings are as follows:
[0190] Table 1. Reinforcement learning hyperparameter settings
[0191]
[0192] The learning rate uses hot start and linear decay techniques. In the first 5% of the total number of training steps, the learning rate increases linearly from 0.01 times the set learning rate to the set learning rate. It then decays linearly with training, with a minimum of 0.01 times the set learning rate. The optimizer uses the AdamW optimizer, with parameters eps=1e-5 and weight_decay (weight decay)=1e-3.
[0193] After training, the prediction accuracy of Model 1 on the difficult sample data set is greatly improved, but it has a forgetting problem, and the prediction accuracy of Model 1 on all samples decreases.
[0194] 3) Reinforcement Learning with Supervised Loss:
[0195] Initialize the model with the weights of model 1, and use reinforcement learning with supervised loss to train the model on the mixed sample dataset to obtain model 2 (i.e., the second model). The reward function and hyperparameter settings used are the same as 2.
[0196] The supervision loss is calculated by the cross entropy loss between the training model output and the sample label, and the total loss is obtained by adding it to the reinforcement learning loss in a certain ratio.
[0197] ;
[0198] in, is the coefficient, and its value is 0.02.
[0199] The prediction accuracy of model 2 on the difficult sample data set is further improved, consolidating the knowledge learned on the difficult sample data set. At the same time, model 2 alleviates the forgetting problem, and the prediction accuracy on all samples is improved compared with model 1, but it is still slightly lower than the supervised model.
[0200] 4) Knowledge Distillation:
[0201] Knowledge distillation includes two models, namely the teacher model and the student model. The purpose of knowledge distillation is to hope that the student model can learn the knowledge of the teacher model, thereby improving the performance of the student model.
[0202] The process of knowledge distillation is as follows Figure 6 As shown in the figure: The student model and the teacher model accept the same feature input and infer their own logits (i.e., probability distribution function). The KL loss is calculated based on the KL divergence of the two logits, and the cross entropy loss between the student model output and the label is calculated as the supervision loss. The total loss is the sum of the KL loss and the supervision loss, and the student network model is optimized based on this loss.
[0203] ;
[0204] The supervised model is used as the student model and model 2 is used as the teacher model for training to obtain model 3 (i.e. the enhanced code generation model). The optimizer uses the AdamW optimizer with default parameters. The hyperparameter settings are shown in the following table.
[0205] Table 2 Knowledge sorting hyperparameter settings
[0206]
[0207] Experiments have shown that most indicators of Model 3 on difficult sample datasets and all datasets are higher than those of the supervised model. The experimental results are shown in the following table.
[0208] Table 3 Experimental results
[0209]
[0210] In this embodiment, a reinforcement learning environment, a reinforcement learning model, and reinforcement learning rewards are constructed, and reinforcement learning training with static feedback is performed according to the reinforcement learning algorithm. The set reinforcement rewards are used for continuous optimization, thereby improving the prediction accuracy of the supervision model.
[0211] This embodiment achieves the following technical effects through the above technical solution:
[0212] This embodiment uses reinforcement learning with feedback from a static type checker to fine-tune the generative model. In the model trained by supervised learning, type predictions of variables in the software can be obtained and passed to the static type checker to see if there are any type conflicts. Then, reinforcement learning with a specific reward function and training algorithm is used to fine-tune the generative model to avoid knowledge of type errors; this embodiment proposes using a generation-based method to predict types, and using reinforcement learning to enhance the accuracy of type predictions of the generative model, solving the problem that the prediction results of the existing type inference model deviate greatly from actual applications.
[0213] Exemplary Devices
[0214] Based on the above embodiments, the present invention further provides a type inference device based on reinforcement learning, comprising:
[0215] A preprocessing module, used for acquiring operating environment data and code data, and preprocessing the operating environment data and the code data to obtain a legal type sequence and a type flow sequence;
[0216] A supervised training module, used for performing supervised training of a code generation task with type annotations according to the legal type sequence and the type flow sequence;
[0217] The reinforcement learning module is used to perform static feedback reinforcement learning training based on the generated target code with type annotations to obtain a reinforced code generation model.
[0218] Based on the above embodiments, the present invention also provides a computer terminal, comprising: a processor, a memory, an interface, a display screen and a communication module connected via a system bus; wherein the processor is used to provide computing and control capabilities; the memory includes a storage medium and an internal memory; the storage medium stores an operating system and a computer program; the internal memory provides an environment for the operation of the operating system and the computer program in the storage medium; the interface is used to connect external devices, such as mobile terminals and computers; the display screen is used to display corresponding information; and the communication module is used to communicate with a cloud server or a mobile terminal.
[0219] When the computer program is executed by the processor, it is used to implement the operation of a type inference method based on reinforcement learning.
[0220] It can be understood by those skilled in the art that Figure 7 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the computer terminal to which the scheme of the present invention is applied. The specific computer terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0221] In one embodiment, a computer terminal is provided, comprising: a processor and a memory, wherein the memory stores a reinforcement learning-based type inference program, and the reinforcement learning-based type inference program is used to implement the operation of the reinforcement learning-based type inference method described above when executed by the processor.
[0222] In one embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a reinforcement learning-based type inference program, and when the reinforcement learning-based type inference program is executed by the processor, it is used to implement the operation of the reinforcement learning-based type inference method as described above.
[0223] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory.
[0224] In summary, the present invention provides a type inference method and device based on reinforcement learning, the method comprising: obtaining operating environment data and code data, and preprocessing the operating environment data and the code data to obtain a legal type sequence and a type flow sequence; performing supervised training of a code generation task with type annotations according to the legal type sequence and the type flow sequence; performing reinforcement learning training with static feedback according to the generated target code with type annotations to obtain a reinforced code generation model. The present invention proposes using a generation-based method to predict types, and enhancing the accuracy of type prediction of the generation model through reinforcement learning, thereby solving the problem that the prediction results of the existing type inference model deviate greatly from the actual application.
[0225] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.< / eol>
Claims
1. A type inference method based on reinforcement learning, It is characterized in that include: Acquire operating environment data and code data, and preprocess the operating environment data and the code data to obtain a legal type sequence and a type stream sequence; Performing supervised training of a code generation task with type annotations according to the legal type sequence and the type flow sequence; Based on the generated target code with type annotations, static feedback reinforcement learning training is performed to obtain a reinforced code generation model; The supervised training of the code generation task with type annotations according to the legal type sequence and the type flow sequence includes: Generate a training data set according to the legal type sequence, the type flow sequence, and a public code example with type annotations; Perform supervised training of a code generation task with type annotations on an encoder and a decoder according to the training data set to obtain a supervised model; The static feedback reinforcement learning training is performed according to the generated target code with type annotation to obtain the enhanced code generation model, which includes: Based on the labeled data, a reinforcement learning environment is constructed for reinforcement learning training; the reinforcement learning environment includes: the state and action of the reinforcement learning environment and the environment operation process of the reinforcement learning environment; Constructing a reinforcement learning model, and setting an end-of-round reward and a relative entropy reward based on the reinforcement learning model; The method of performing static feedback reinforcement learning training according to the generated target code with type annotation to obtain a reinforced code generation model includes: Aggregate samples that are predicted incorrectly by the supervised model into a difficult sample dataset; Sampling common samples of the same size as the difficult samples from all samples, and combining the difficult sample data set and the common samples to form a mixed data set; Initialize the model using the weights of the supervised model, and train the supervised model on the difficult sample data set using a reinforcement learning algorithm to obtain a first model; Initializing the model with the weights of the first model, and training the model on the mixed data set using reinforcement learning with supervised loss to obtain a second model; The supervision model is used as a student model and the second model is used as a teacher model for training to obtain the enhanced code generation model; The supervised loss of the reinforcement learning with supervised loss is: calculating the cross entropy loss between the training model output and the sample label, and adding the calculated cross entropy loss to the reinforcement learning loss in a certain ratio to obtain the total loss; The reinforcement learning algorithm is a proximal strategy optimization algorithm. The reinforcement learning algorithm uses an actor-critic architecture. The network model of the reinforcement learning algorithm includes: a strategy output layer and a value output layer. The strategy output layer is used to output the probability distribution of the actions that the agent can perform in the current state, and the value output layer is used to output the value of the current state. The code generation model is a sequence model based on an encoder-decoder architecture, which is used to perform code generation tasks with type annotations. The code generation model accepts a legal type sequence parsed by a runtime environment and an abstract syntax tree of a target code as input, and generates a target code with type annotations.
2. The type inference method based on reinforcement learning according to claim 1, It is characterized in that The preprocessing of the operating environment data and the code data includes: Perform runtime environment dependency inference to obtain the corresponding third-party package versions; Based on the defined user types and / or the user types imported by the third-party package version, analyze to obtain all user-defined types and obtain the legal type sequence; Data flow analysis is performed on the abstract syntax tree of the target function to obtain all data flows, and all data flows are screened to obtain the type flow sequence.
3. A type inference device based on reinforcement learning, used to implement the type inference method based on reinforcement learning as described in any one of claims 1-2, It is characterized in that include: A preprocessing module, used for acquiring operating environment data and code data, and preprocessing the operating environment data and the code data to obtain a legal type sequence and a type flow sequence; A supervised training module, used for performing supervised training of a code generation task with type annotations according to the legal type sequence and the type flow sequence; The reinforcement learning module is used to perform static feedback reinforcement learning training based on the generated target code with type annotations to obtain a reinforced code generation model.
4. A computer terminal, It is characterized in that include: A processor and a memory, wherein the memory stores a type inference program based on reinforcement learning, and when the type inference program based on reinforcement learning is executed by the processor, it is used to implement the operation of the type inference method based on reinforcement learning as described in any one of claims 1-2.
5. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a reinforcement learning-based type inference program, which, when executed by a processor, is used to implement the operation of the reinforcement learning-based type inference method as described in any one of claims 1-2.
