An Encoding Dynamic Completion Method for Large Language Models

Abstract

The present invention proposes a coding dynamic completion method for large language models. The method includes: constructing a mixed training dataset; initializing a Transformer model architecture that integrates probability space topological transformation; performing model training with collaborative optimization of the main objective and the auxiliary objective based on the mixed training dataset; at each decoding step, the model calculates the probability distribution of the next token according to the current context; constraining the generated probability distribution within the effective probability subspace through spherical probability mapping, and combining with the taboo table mechanism to filter syntax error candidates in real time and adjust the probability distribution; dynamically selecting a sampling strategy according to the current decoding depth, sampling from the adjusted probability distribution to obtain the next code snippet, and finally generating code completion suggestions that conform to the abstract syntax tree rules and semantic constraints. By combining the probability distribution constraint and the taboo table mechanism, the model of the present invention can dynamically adjust the generated probability distribution, enabling more appropriate candidate tokens to obtain higher probabilities, thereby improving the accuracy of code completion.

Description

Technical Field

[0001] The present invention relates to the technical field of code completion, and particularly to a coding dynamic completion method for large language models. Background Art

[0002] In the process of software development, code completion technology can help programmers quickly and accurately complete code snippets, so as to improve programming efficiency and reduce programming errors, thereby accelerating the development cycle and reducing the error rate.

[0003] Traditional code completion technologies often rely on predefined code rules or templates, and generate completion suggestions by matching the current context with patterns in the rule library. However, such completion methods have great limitations: due to the fixity of the rules or templates, it is difficult to adapt to the diversity and variability in the code, and complex or unseen code patterns cannot be processed; with the continuous update of programming languages and frameworks, the rule library needs to be frequently updated and maintained, increasing the complexity and cost of technical implementation.

[0004] Currently, some code completion methods based on deep learning have also emerged, such as RNN, LSTM, Transformer, etc. These methods use deep learning models to learn complex patterns and dependencies in the code and can generate more accurate code completion suggestions. But there are also some limitations as follows: it is often difficult to ensure compliance with syntax rules and semantic constraints when generating code, resulting in syntax errors or semantic inconsistencies in the generated code; during the decoding process, the probability distribution generated by the model may contain a large number of candidate tokens that do not conform to syntax or semantics, lacking an effective mechanism to constrain and optimize its probability distribution, reducing the accuracy of code completion. Summary of the Invention

[0005] Based on this, the purpose of the present invention is to propose a coding dynamic completion method for large language models to solve the above-mentioned problems.

[0006] A coding dynamic completion method for large language models according to the present invention, the method includes:

[0007] Construct a mixed training dataset, including multi-language code snippets, syntax constraint annotations, and domain pre-training corpus;

[0008] Initialize the Transformer model architecture that fuses probability space topological transformation, and define the hyper-sphere dimension and dynamic effective region parameters;

[0009] Based on the mixed training dataset, perform model training with collaborative optimization of the main objective and the auxiliary objective, where the main objective is the code completion accuracy rate, and the auxiliary objectives include syntax tree validity prediction and semantic constraint satisfaction discrimination;

[0010] At each decoding step, the model calculates the probability distribution of the next token based on the current context;

[0011] During the model decoding phase, the generated probability distribution is constrained within the valid probability subspace through spherical probability mapping. Combining with the taboo list mechanism, syntax error candidates are filtered in real-time and the probability distribution is adjusted;

[0012] According to the current decoding depth, the sampling strategy is dynamically selected. Based on the selected sampling strategy, the next code snippet is sampled from the adjusted probability distribution, and finally, code completion suggestions that conform to the abstract syntax tree rules and semantic constraints are generated.

[0013] Furthermore, the steps of initializing the Transformer model architecture with probability space topology transformation include:

[0014] Insert a probability space transformation module into the standard Transformer architecture;

[0015] Define the parameters of the hypersphere probability space;

[0016] Set the dynamic valid region parameters and constrain the generated probability distribution through the spherical mapping function;

[0017] Load the domain pre-trained weights when initializing the model parameters.

[0018] Furthermore, the steps of, at each decoding step, the model calculating the probability distribution of the next token based on the currently generated code sequence include:

[0019] Convert the currently generated code sequence into an embedding vector;

[0020] Through the masked self-attention mechanism, capture the internal dependencies of the currently generated code sequence;

[0021] Use the output of the self-attention layer as the query, and the output of the encoder as the key and value;

[0022] Integrate the output of the encoder and the information of the currently generated code sequence, perform a non-linear transformation on the integrated information, and generate the final output representation;

[0023] Map the final output representation to the space of the vocabulary size through a linear layer;

[0024] Use the Softmax function to generate the prediction probability distribution of each candidate token in the output sequence as the next token, and use the set of all candidate tokens in the output sequence as the candidate sequence.

[0025] Further, the step of constraining the generated probability distribution within the effective probability subspace through spherical probability mapping and combining with the taboo list mechanism to filter syntax error candidates in real time and adjust the probability distribution includes:

[0026] Map the generated probability distribution to the unit sphere and achieve uniform constraint of the probability distribution through octahedral parameterization;

[0027] Create a taboo list to record the error patterns and their probability characteristics generated during the historical decoding process;

[0028] At each step of decoding, query the taboo list according to the current candidate sequence to filter known error patterns, where the candidate sequence is the sequence composed of all possible tokens generated by the model for the next token position;

[0029] Recalculate the spherical probability for the filtered candidate sequence to increase the generation probability of valid candidate tokens;

[0030] Dynamically adjust the spherical mapping parameters and taboo list update strategy according to the decoding progress.

[0031] Further, the step of mapping the generated probability distribution to the unit sphere and achieving uniform constraint of the probability distribution through octahedral parameterization includes:

[0032] Convert the high-dimensional vector representation of each generated candidate token to spherical coordinates through the octahedral mapping formula. The mapping formula is: |x| + |y| + |z| = 1, where x, y, and z are the coordinates of the generated candidate token in three-dimensional space respectively;

[0033] Normalize the mapped coordinates to ensure that all candidate tokens are located on the unit sphere;

[0034] Calculate the probability value of each candidate token through the spherical coordinate system so that the probability distribution satisfies the uniformity and effectiveness of the spherical space.

[0035] Further, the step of querying the taboo list according to the current candidate sequence at each step of decoding to filter known error patterns includes:

[0036] Conduct a syntax check on the currently generated candidate sequence and extract the error patterns;

[0037] Query the taboo list. If the candidate sequence contains the error patterns in the taboo list, filter it out;

[0038] Update the filtered candidate sequence.

[0039] Further, the step of recalculating the spherical probability for the filtered candidate sequences to enhance the generation probability of valid candidate tokens includes:

[0040] Resampling the filtered candidate sequences through the spherical probability mapping method;

[0041] Calculating the spherical probability values of each valid candidate token and adjusting the probability distribution according to the spherical probability values, such that the adjusted probability distribution is more concentrated on the valid candidates.

[0042] Further, the step of dynamically adjusting the spherical mapping parameters and the taboo table update strategy according to the decoding progress includes:

[0043] Monitoring the code generation quality and syntax validity during the decoding process and adjusting the spherical mapping parameters in real time;

[0044] Dynamically updating the taboo length and capacity of the taboo table according to the filling situation of the taboo table and the frequency of error patterns;

[0045] Through a reinforcement learning mechanism, giving higher taboo priorities to the candidates that lead to error patterns and optimizing the self-update strategy of the taboo table.

[0046] Further, the step of dynamically updating the taboo length and capacity of the taboo table according to the filling situation of the taboo table and the frequency of error patterns includes:

[0047] Setting an optimal interval for the taboo length and randomly selecting a taboo length from the optimal interval for the taboo length as the initial taboo length;

[0048] If the optimal solution formed by combining the optimal candidate token in the current candidate sequence with the currently generated code sequence is superior to the currently recorded optimal solution in terms of the evaluation criteria, then increase the taboo length;

[0049] Otherwise, decrease the taboo length;

[0050] Dynamically adjusting the capacity of the taboo table according to the frequency of error patterns to ensure that the taboo table can effectively store and update error patterns.

[0051] Further, the step of dynamically selecting a sampling strategy according to the current decoding depth, sampling the next code snippet from the adjusted probability distribution based on the selected sampling strategy, and finally generating a code completion suggestion that conforms to the abstract syntax tree rules and semantic constraints includes:

[0052] During the code completion process, monitoring the length of the generated code sequence in real time to judge the current decoding depth;

[0053] When the current decoding depth is relatively shallow, the Top-k sampling strategy is adopted, and the k code segments with the highest probabilities in the probability distribution are selected as candidates;

[0054] When the current decoding depth is relatively deep, the temperature sampling strategy is adopted to control the randomness of the sampling process by adjusting the temperature parameter;

[0055] Based on the selected sampling strategy, the next code segment is sampled from the adjusted probability distribution;

[0056] The sampled code segment is added to the generated code sequence to form a code completion suggestion that conforms to the abstract syntax tree rules and semantic constraints.

[0057] In summary, the encoding dynamic completion method for large language models of the present invention integrates multi-language code segments, explicit syntax constraint annotations, and domain-specific pre-trained corpora to construct a mixed training set. The syntax constraint annotations provide the model with clear syntax rule information, enabling the model to better understand the syntax structure of the code during the learning process. The domain pre-trained corpora help the model master the semantic knowledge of specific domains, so that more domain-semantic-compliant codes can be generated during code completion; by introducing hypersphere dimension constraints and dynamic effective region parameters in the Transformer, the generated probability distribution is restricted within the syntax valid subspace, which can more accurately constrain the generated probability distribution and improve the accuracy of code completion; a collaborative optimization framework of the main objective (i.e., completion accuracy) and the secondary objectives (including syntax tree validity prediction and semantic constraint satisfaction discrimination) is established to guide the model to not only focus on code completion itself during training, but also pay attention to whether the generated code conforms to syntax rules and semantic constraints, thereby enhancing the model's satisfaction with syntax and semantic constraints; at each decoding step, the model aggregates the semantic vectors of the current context and generates an accurate probability distribution through the self-attention mechanism; and after generating the probability distribution at each step, by converting the discrete token probability into a continuous probability flow optimization problem through spherical probability mapping and combining with the taboo list mechanism, the model can dynamically adjust the generated probability distribution, making more suitable candidate tokens obtain higher probabilities, thereby improving the accuracy of code completion; the sampling strategy is dynamically adjusted according to the decoding depth, enabling the model to flexibly select the most appropriate sampling strategy according to different decoding stages, thereby effectively balancing the problem of generation efficiency and quality and significantly improving the compilation passing rate of long code segments.

[0058] Additional aspects and advantages of the present invention will be given in part in the following description, will become apparent in part from the following description, or will be understood through the embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The above and / or additional aspects and advantages of the present invention will become apparent and easy to understand from the description of the embodiments in conjunction with the following drawings, where:

[0060] Figure 1 This is a flowchart of a method for encoding dynamic completion for large language models according to Embodiment 1 of the present invention. Detailed implementation manner

[0061] To facilitate the understanding of the present invention, the present invention will be described more comprehensively below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, these embodiments are provided to make the disclosure of the present invention more thorough and comprehensive.

[0062] It should be noted that when an element is referred to as being "fixed to" another element, it can be directly on the other element or there may also be a middle element. When an element is considered to be "connected" to another element, it can be directly connected to the other element or there may be a middle element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are only for the purpose of illustration.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs. The terms used in the description of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items. Embodiment

[0064] Please refer to Figure 1 , the present invention proposes a method for encoding dynamic completion for large language models, and the method includes steps S101 to S106:

[0065] S101, constructing a mixed training dataset, including multilingual code snippets, syntax constraint annotations, and domain pre-training corpora.

[0066] Fusing multilingual code snippets, explicit syntax constraint annotations, and domain-specific pre-training corpora to construct a mixed training set, which enables the model to have the ability to understand multilingual paradigms and master domain-specific syntax rules (such as Python indentation, Java type declarations) at the same time, and can significantly improve the adaptability of the model in different programming languages, thereby solving the semantic drift problem of traditional single-modal models in cross-domain completion.

[0067] Further optionally, the step of constructing the mixed training dataset includes:

[0068] Collecting open-source code libraries in different languages from multiple sources;

[0069] Use a code parsing tool or library to parse the collected code into an abstract syntax tree structure;

[0070] Utilize a static code analysis tool to annotate the parsed abstract syntax tree, extract syntactic constraint features, including variable types, function parameters, syntactic rule compliance, and potential issues;

[0071] Fuse the code pattern data in the domain pre-training corpus with the annotated code;

[0072] Generate mixed training samples containing syntactic labels and semantic features based on the annotated syntactic features and fused code pattern data.

[0073] Specifically, code in different languages can be collected from multiple open-source code platforms. For example, collect project codes of the Python web development framework Django, the Java enterprise application development framework Spring Boot, and the JavaScript front-end framework React from GitHub. These projects cover applications in different fields and of different scales, with rich code examples.

[0074] Use Python's abstract syntax tree library to parse Python code. After parsing, its abstract syntax tree structure can be obtained, including function definition nodes, parameter nodes, return statement nodes, etc.

[0075] For the parsed abstract syntax tree, use a static code analysis tool for annotation. Extract syntactic constraint features from the output of the static code analysis tool. For example, for the following add function code, the extracted features include the function name add, the types of parameters a and b (assumed to be integers by inferring from comments or context), the return value type of the function as an integer, and the syntactic rule compliance (correct indentation), etc. Code example:

[0076] def add(a, b):

[0077] return a + b

[0078] Extract code pattern data from the pre-training corpus. For example, extract the code pattern for handling GET requests:

[0079] from flask import Flask, request

[0080] app = Flask(__name__)

[0081] @app.route(' / get_data', methods=['GET'])

[0082] def get_data():

[0083] data = request.args.get('param')

[0084] return data

[0085] The method of fine - tuning the pre - trained model is used to fuse the labeled code with the code pattern data extracted from the domain pre - trained corpus. That is, the labeled code and the code pattern data are used as inputs to fine - tune the pre - trained model. During the fine - tuning process, the model is allowed to learn information such as the syntactic features in the code and domain - specific code patterns. So that the fused data can fully reflect the code features within the domain.

[0086] According to the labeled syntactic features and the fused code pattern data, mixed training samples containing syntactic labels and semantic features are generated. For example, for the add function and the code for handling GET requests, a training sample is generated, which contains syntactic labels of the function definition (such as function name, parameter types, return value type), semantic features of the code (such as the function's function is to add, and the function of handling GET requests is to obtain parameters and return), etc. The sample is as follows:

[0087] {

[0088] "code": "def add(a, b):\n return a + b",

[0089] "language": "Python",

[0090] "syntax_labels": {

[0091] "function_name": "add",

[0092] "parameters": ["a", "b"],

[0093] "return_type": "int"

[0094] },

[0095] "semantic_features": {

[0096] "function_purpose": "Add two numbers"

[0097] }

[0098] },

[0099] {

[0100] "code": "from flask import Flask, request\n\napp = Flask(__name__)\n\n@app.route(' / get_data', methods=['GET'])\ndef get_data():\n data =request.args.get('param')\n return data",

[0101] "language": "Python",

[0102] "syntax_labels": {

[0103] "framework": "Flask",

[0104] "route": " / get_data",

[0105] "method": "GET"

[0106] },

[0107] "semantic_features": {

[0108] "function_purpose": "Handle GET request and return parameter value"

[0109] }

[0110] }

[0111] Thus, a multi-modal hybrid training dataset containing multi-language code snippets, syntax constraint annotations, and domain pre-training corpora is constructed.

[0112] S102. Initialize the Transformer model architecture for fusing probability space topology transformation, and define the hyper-sphere dimension and dynamic effective region parameters.

[0113] Introduce hyper-sphere probability space constraints in the Transformer to restrict the generation process to the syntactically valid sub-manifold, making the code generated by the model more compliant with syntax rules. By dynamically adjusting the effective region parameters, the model can maintain an exploratory nature (wide probability distribution) at the initial stage of generation, enabling it to explore more code generation possibilities and increasing the diversity of the generated code; gradually shrink to legal candidates (narrow probability distribution) in the later stage to avoid generating code that does not conform to the syntax, thereby improving the syntactic correctness of the generated code.

[0114] Further optionally, the steps of initializing the Transformer model architecture for fusing probability space topology transformation include:

[0115] Insert a probability space transformation module into the standard Transformer architecture;

[0116] Define hyper-sphere probability space parameters;

[0117] Set the dynamic effective region parameters and generate a probability distribution through a spherical mapping function;

[0118] Load domain pre-trained weights when initializing the model parameters.

[0119] Specifically, insert a probability space transformation module after the decoder layer of the Transformer. The probability space transformation module includes a spherical mapping layer and a dynamic effective region adjustment mechanism;

[0120] Set the dimension N of the probability space hyper-sphere and define the radius of the standardized hyper-sphere surface;

[0121] Randomly sample an initial center vector on the hyper-sphere surface, and use orthogonality constraints to ensure ;

[0122] Set the initial effective radius , and define the effective region as a hyper-spherical cap region centered at with a radius of ;

[0123] Define an effective region expansion coefficient α ∈ [0.9, 1.1], and set the update rule of the effective region radius as to achieve dynamic adjustment of the effective region;

[0124] Construct a spherical mapping function. When v ≠ 0, ; when v = 0, ;

[0125] Map the probability vector output by the model to the hyper-sphere through the spherical mapping function to achieve probability space transformation;

[0126] Load pre-trained parameters from the standard GPT model and freeze the parameters of its first L - 1 layers, where L is the total number of layers of the GPT model;

[0127] Perform Xavier initialization on the newly added parameters while keeping the GPT parameters unchanged. The newly added parameters include the center vector c0 and the effective radius , and the GPT parameters include the weight matrix W and the bias term b;

[0128] Implement parameter constraints after each parameter update to ensure that the center vector and the effective radius are within a reasonable range.

[0129] S103. Based on the mixed training dataset, perform model training for the collaborative optimization of the primary objective and the secondary objectives, where the primary objective is the code completion accuracy, and the secondary objectives include the prediction of the validity of the syntax tree and the discrimination of the satisfaction degree of semantic constraints.

[0130] Adopt a multi-task learning framework, where the primary task (completion accuracy) and the secondary tasks (prediction of the validity of the syntax tree, discrimination of semantic constraints) form complementary supervision. For example, when the model generates candidates with high probabilities but syntax errors, the secondary tasks will backpropagate constraint signals, forcing the model to recalibrate the probability distribution, and finally, the passing rate of the generated code under the verification of the abstract syntax tree parser is significantly improved.

[0131] Further optionally, the steps of performing model training for the collaborative optimization of the primary objective and the secondary objectives based on the mixed training dataset include:

[0132] Define a joint loss function, which is composed of the weighted cross-entropy loss of the code completion task, the binary cross-entropy loss of the syntax tree validity prediction task, and the binary cross-entropy loss of the semantic constraint satisfaction discrimination task.

[0133] Through the backpropagation algorithm and the adaptive optimizer, iteratively optimize the model parameters on the mixed training dataset to achieve the collaborative optimization of the primary objective and the secondary objectives.

[0134] Specifically, when training the model, first preprocess the mixed training dataset, including operations such as data cleaning, tokenization, and encoding. The mixed training dataset includes multi-language code snippets, syntax constraint annotations, and domain pre-training corpora.

[0135] Then, calculate the cross-entropy loss between the code completion results predicted by the model and the true code, which is used to measure the prediction accuracy of the model in the code completion task and is the core optimization objective of the code completion scheme. For the syntax tree validity prediction task, calculate the binary cross-entropy loss between the syntax tree validity labels predicted by the model and the true labels to evaluate the model's ability to judge the validity of the syntax tree and assist the code completion scheme in generating code that conforms to the syntax rules. For the semantic constraint satisfaction discrimination task, calculate the binary cross-entropy loss between the semantic constraint satisfaction labels predicted by the model and the true labels to measure the model's discrimination ability of the degree of satisfaction of semantic constraints and ensure that the generated code by the code completion scheme is semantically reasonable and consistent. Combine the loss functions of the above three tasks with a certain proportion of weighting to form a joint loss function. During the training process, dynamically adjust the weights of each task in the joint loss function according to the performance of the model on each task. For example, when the performance of the model on a certain task improves rapidly, the weight of this task can be appropriately reduced to encourage the model to achieve better performance on other tasks.

[0136] Input the preprocessed mixed training dataset into the initialized model for forward propagation calculation to obtain the prediction results on each task (including code completion results, syntax tree validity labels, semantic constraint satisfaction labels). And calculate the total loss of the model on the current batch of data according to the joint loss function. Then, through the backpropagation algorithm, calculate the gradient of the loss function with respect to the model parameters. Use an adaptive optimizer to update the model parameters according to the calculated gradient to minimize the joint loss function.

[0137] Repeat the above steps of forward propagation, calculating the joint loss, backpropagation, and parameter update until reaching the preset number of training epochs or meeting other stopping conditions.

[0138] S104. At each decoding step, the model calculates the probability distribution of the next token based on the current context.

[0139] In the decoding stage, use the trained model to gradually calculate the probability distribution of the next token at each decoding step based on the current context. Specifically, the model will receive the currently generated token sequence (i.e., the context) and predict the next possible token based on the language patterns and probability distributions learned internally. This prediction process will be iterated until a complete text sequence is generated or the preset stopping condition is reached.

[0140] The probability distribution generated by the model is the basis for subsequent spherical probability mapping and taboo list mechanism operations. Subsequently, the generated probability distribution can be constrained within the effective probability subspace through spherical probability mapping, and combined with the taboo list mechanism to filter syntax error candidates in real time and adjust the probability distribution, further improving the quality and syntax correctness of the generated code.

[0141] Further optionally, the step of the model calculating the probability distribution of the next token according to the currently generated code sequence at each decoding step includes:

[0142] Convert the currently generated code sequence into an embedding vector;

[0143] Through the masked self-attention mechanism, capture the internal dependencies of the currently generated code sequence;

[0144] Use the output of the self-attention layer as the query, and the output of the encoder as the key and value;

[0145] Integrate the output of the encoder and the information of the currently generated code sequence, perform a non-linear transformation on the integrated information, and generate the final output representation;

[0146] Map the final output representation to the space of the vocabulary size through a linear layer to associate the output of the model with all possible code tokens;

[0147] Use the Softmax function to generate the predicted probability distribution of each candidate token in the output sequence as the next token, and use the set of all candidate tokens in the output sequence as the candidate sequence.

[0148] Specifically, convert each token of the currently generated code sequence into its corresponding word embedding vector. In this process, each token is represented as a high-dimensional vector.

[0149] Through the self-attention mechanism, the model is allowed to consider all tokens in the currently generated sequence when generating the next token. The masked self-attention mechanism can ensure that when generating the t-th token, the model can only access the information of the 1st to the t-1-th tokens. In this process, the model will calculate the attention weights between each token and other tokens, and these weights reflect the dependencies between tokens.

[0150] In the decoder of the Transformer model, use the output of the self-attention layer as the query, and the output of the encoder as the key and value, so as to combine the information of the encoder (key and value) and the output of the decoder self-attention layer (query). Enable the model to integrate information from the encoder and decoder, and more comprehensively understand the input sequence and the generated sequence.

[0151] Then, integrate the output of the encoder and the information of the currently generated code sequence, perform a non-linear transformation on the integrated information to generate a final output representation, that is, a final high-dimensional vector representation, which captures the complex relationship between the input sequence and the generated sequence.

[0152] Map the final output representation to a space of the vocabulary size through a linear layer, that is, map the high-dimensional vector representation of the final output representation to a vector space with the same size as the vocabulary. Each dimension corresponds to a token in the vocabulary, and each value in the mapped vector represents the score or probability of the corresponding token as the next token.

[0153] Then use the Softmax function to generate the predicted probability distribution of each candidate token in the output sequence as the next token, so that the sum of the values of all dimensions is 1, and each value of each dimension represents the predicted probability of the corresponding token as the next token. And take the set of all candidate tokens in the output sequence as the candidate sequence:

[0154] S105, in the model decoding stage, constrain the generated probability distribution within the valid probability subspace through spherical probability mapping, and combine the taboo list mechanism to filter out syntax error candidates in real time and adjust the probability distribution.

[0155] At each step of model decoding, based on the probability distribution generated by the model, constrain the generated probability distribution within the valid probability subspace through spherical probability mapping, and combine the taboo list mechanism to filter out syntax error candidates in real time and adjust the probability distribution, further improving the quality and syntax correctness of the generated code.

[0156] Further optionally, the step of constraining the generated probability distribution within the valid probability subspace through spherical probability mapping, combining the taboo list mechanism, filtering out syntax error candidates in real time and adjusting the probability distribution includes:

[0157] Map the generated probability distribution to the unit sphere and achieve uniform constraint of the probability distribution through octahedral parameterization;

[0158] Create a taboo list to record the error patterns and their probability characteristics generated during the historical decoding process;

[0159] At each step of decoding, query the taboo list according to the current candidate sequence to filter out known error patterns;

[0160] Recalculate the spherical probability for the filtered candidate sequence to increase the generation probability of valid candidate tokens;

[0161] Dynamically adjust the spherical mapping parameters and the taboo list update strategy according to the decoding progress.

[0162] It is understandable that the primary task of the code completion system is to generate syntactically correct code. Traditional models generate code snippets with syntax errors during the decoding process, such as unclosed parentheses, type mismatches, etc., resulting in compilation failures or runtime errors. By means of spherical probability mapping, restricting the generated probability distribution within the subspace of syntactic validity can significantly reduce the probability of generating code with syntax errors. Specifically, the probability distribution can be mapped to the unit sphere through spherical probability mapping, and uniform constraints can be achieved using methods such as octahedral parameterization, enabling the model to be more inclined to select syntactically correct candidate tokens. This constraint can ensure the syntactic correctness of the generated code, improving the compilability and executability of the code.

[0163] During the decoding process, dynamically identify and filter out candidate tokens that may cause syntax errors, thereby further improving the quality of the generated code. The taboo list mechanism is used to record the error patterns and their probability characteristics that occurred in the historical decoding process. At each step of decoding, the model filters out those candidate sequences that are known to cause syntax errors by querying the taboo list. This real-time filtering mechanism can effectively avoid repeating the same syntax errors, improving the accuracy and reliability of the generated code.

[0164] After filtering out the syntax error candidates, readjust the probability distribution so that the remaining valid candidate tokens have a higher generation probability, thereby optimizing the generation result. Specifically, the spherical probability of the filtered candidate sequence can be recalculated, and the probability distribution can be adjusted according to the spherical probability value. This adjustment can ensure that the model is more inclined to select those candidate tokens that are syntactically correct and semantically reasonable when generating code, improving the quality and practicality of the generated code.

[0165] While ensuring the syntactic correctness of the generated code, maintain the diversity of the generation, and avoid over-constraint resulting in overly single or lack of innovation in the generated code. Specifically, by dynamically adjusting the spherical mapping parameters and the taboo list update strategy, the model can adjust the strictness of the constraint in real time according to the decoding progress and the quality of the generated code. For example, a relatively loose constraint can be maintained at the initial stage of decoding to encourage generation diversity; while at the later stage of decoding, the constraint can be gradually tightened to ensure the syntactic correctness and semantic consistency of the generated code. This dynamic adjustment mechanism enables the model to find a balance between syntactic constraints and generation diversity.

[0166] Further optionally, the step of mapping the generated probability distribution to the unit sphere and achieving uniform constraint of the probability distribution through octahedral parameterization includes:

[0167] Convert the high-dimensional vector representation of each generated candidate token into spherical coordinates through the octahedral mapping formula. The mapping formula is: |x| + |y| + |z| = 1, where x, y, and z are the coordinates of the generated candidate token in three-dimensional space respectively;

[0168] Normalize the mapped coordinates to ensure that all candidate tokens lie on the unit sphere;

[0169] Calculate the probability value of each candidate token through the spherical coordinate system, so that the probability distribution satisfies the uniformity and effectiveness of the spherical space.

[0170] In this embodiment, the high-dimensional vector representation of the generated candidate tokens is converted into spherical coordinates through the octahedron mapping formula, and the probability value of each candidate token is calculated, so as to realize the uniform constraint of the probability distribution. The uniform constraint can ensure that the generated probability distribution is smoother and more reasonable, thereby improving the quality of the generated code.

[0171] This is illustrated by the following example:

[0172] Suppose that in a certain decoding step, the model generates three candidate tokens for the next token: A, B, C, and the original probability distribution is: 0.2, 0.3, 0.5. Each token is represented as a high-dimensional vector inside the model. For example, Token A: the vector representation is (0.5, 0.5, 0.5), Token B: the vector representation is (-0.5, 0.5, 0), Token C: the vector representation is (0, -0.5, 0.5).

[0173] Map the above three-dimensional vectors to the surface of the unit octahedron through octahedron mapping and approximately satisfy spherical coordinates. The octahedron mapping formula is |x| + |y| + |z| = 1. Direct application of this formula may require scaling and adjustment of the vectors.

[0174] Token A is mapped to (0.267, 0.267, 0.466) (satisfying |0.267| + |0.267| + |0.466| ≈ 1) through scaling and adjustment; Token B is mapped to (-0.267, 0.267, 0.466); Token C is mapped to (0, -0.267, 0.733).

[0175] The mapped coordinates approximately satisfy the surface of the unit octahedron, but to ensure that they strictly lie on the unit sphere (the modulus length is 1), normalization is performed. The normalization formula is: normalized coordinate = mapped coordinate / ||mapped coordinate||, where ||mapped coordinate|| is the modulus length of the mapped coordinate.

[0176] In the spherical coordinate system, the probability values of candidate tokens are calculated based on their positions. It can be calculated based on the proportion of the area occupied by candidate tokens on the sphere, obtaining a probability of 0.4 for Token A, 0.3 for Token B, and 0.3 for Token C.

[0177] Through the above steps, the high-dimensional vector representation of the generated candidate tokens is converted into spherical coordinates through the octahedron mapping formula, and the probability value of each candidate token is calculated, thus realizing the uniform constraint of the probability distribution.

[0178] Further optionally, the step of querying the taboo table according to the current candidate sequence and filtering known error patterns at each decoding step includes:

[0179] Perform a syntax check on the currently generated candidate sequence and extract error patterns;

[0180] Query the taboo table. If the candidate sequence contains an error pattern in the taboo table, filter it out;

[0181] Update the filtered candidate sequence.

[0182] It is understandable that at each step of decoding, it is first necessary to identify possible syntax errors in the currently generated candidate sequence. Specifically, each candidate token in the candidate sequence can be combined with the currently generated code sequence in turn to simulate the possible state of the code sequence if the candidate token is selected, which can more comprehensively evaluate the rationality of the candidate token. For each candidate token in the candidate sequence, it is added to the next token position (the token position to be generated) of the currently generated code sequence in turn to form a new code sequence combination.

[0183] The taboo table stores known error patterns that occurred during the historical decoding process. By querying the taboo table, code sequence combinations containing error patterns can be identified. Specifically, each new code sequence combination can be matched with the patterns in the taboo table. If a combination matches an error pattern in the taboo table, the candidate token is marked as potentially causing an error and filtered.

[0184] Remove those candidate tokens marked as potentially causing errors from the candidate sequence. In this way, in subsequent decoding steps, these tokens will no longer be considered, thus avoiding potential errors. By filtering out candidate tokens that may cause errors, it can be ensured that only safe and effective candidate tokens are considered in the subsequent decoding process.

[0185] At each step of decoding, the above filtering steps are repeated until a complete code sequence is generated.

[0186] Further optionally, the step of recalculating the spherical probability for the filtered candidate sequences to enhance the generation probability of valid candidate tokens includes:

[0187] Resampling the filtered candidate sequences by means of a spherical probability mapping method;

[0188] Calculating the spherical probability values of each valid candidate token and adjusting the probability distribution according to the spherical probability values, such that the adjusted probability distribution is more concentrated on the valid candidates.

[0189] It is understandable that, by using a spherical probability mapping method (such as octahedral parameterization), the filtered candidate sequences are remapped onto the unit sphere. Resampling is to re-evaluate the distribution of the remaining candidate sequences based on the spherical probability mapping method after filtering out the invalid candidates, ensuring more accurate subsequent probability calculations.

[0190] In the spherical coordinate system, according to the positions of the valid candidate tokens, calculate the spherical probability values of each filtered valid candidate token, and these probability values quantify the distribution density or importance of the valid candidate tokens in the spherical space.

[0191] Adjust the originally generated probability distribution according to the calculated spherical probability values. The probability values of the valid candidate tokens can be increased, while the probability values of the invalid candidate tokens can be decreased (or kept at zero). The adjusted probability distribution is more concentrated on the valid candidates, thereby improving the quality of the generated code.

[0192] When adjusting the probability distribution, it is necessary to ensure that the adjusted distribution still maintains the uniformity of the spherical space (i.e., the probability density of any region on the sphere is proportional to the area of that region) and validity (i.e., the sum of all probability values is 1, and each probability value is between 0 and 1).

[0193] In this embodiment, by recalculating the spherical probability and adjusting the probability distribution, the generation probability of valid candidate tokens can be significantly enhanced, thereby improving the quality of the generated code. By concentrating on the valid candidates, the model can converge to the optimal solution faster, thus accelerating the decoding process.

[0194] Further optionally, the step of dynamically adjusting the spherical mapping parameters and the taboo table update strategy according to the decoding progress includes:

[0195] Monitoring the code generation quality and syntactic validity during the decoding process and adjusting the spherical mapping parameters in real time;

[0196] Dynamically updating the taboo length and capacity of the taboo table according to the filling status of the taboo table and the frequency of error patterns;

[0197] Through the reinforcement learning mechanism, higher tabu priorities are given to candidates that lead to error patterns, and the self-update strategy of the tabu list is optimized.

[0198] It is understandable that in the code completion or generation task, in order to improve the quality and efficiency of the generated code, the spherical mapping parameters and the tabu list update strategy can be dynamically adjusted according to the decoding progress. Specifically, during the decoding process, the quality and syntactic validity of the generated code are monitored in real time to promptly detect and correct potential problems. And according to the monitoring results, the spherical mapping parameters are adjusted in real time, such as the coefficients of the mapping function, the dimensions of the spherical space, etc., to optimize the probability distribution of the generated code and improve the generation quality. Through the monitoring feedback, the spherical mapping parameters are continuously iteratively optimized until a satisfactory generation effect is achieved.

[0199] The tabu list is used to store the error patterns that occur in the historical decoding process to avoid making the same mistakes repeatedly. By dynamically updating the tabu length and capacity of the tabu list according to the filling situation of the tabu list and the frequency of error patterns, the effectiveness and efficiency of the tabu list can be improved.

[0200] The reinforcement learning mechanism can optimize the decision-making strategy according to the environmental feedback. In the code generation task, the reinforcement learning mechanism can give higher tabu priorities to candidates that lead to error patterns, thereby optimizing the self-update strategy of the tabu list and improving the quality of the generated code.

[0201] Specifically, a reinforcement learning model is designed, and this reinforcement learning model can adjust its decision-making strategy according to feedback signals such as the quality and syntactic validity of the generated code. During the decoding process, when a certain candidate token causes a syntax error, the reinforcement learning model gives a higher tabu priority to this candidate token, that is, it is more likely to add it to the tabu list. By continuously iteratively training the reinforcement learning model and optimizing the self-update strategy of the tabu list, the model can more effectively avoid making the same mistakes repeatedly in the subsequent decoding process.

[0202] In this embodiment, by dynamically adjusting the spherical mapping parameters and the tabu list update strategy, the quality and syntactic validity of the generated code can be significantly improved. The dynamic adjustment strategy enables the model to adapt to different decoding progress and generation environments, improving the robustness and flexibility of the model. And by optimizing the self-update strategy of the tabu list, the model can converge to the optimal solution faster, reducing unnecessary exploration and trial-and-error processes.

[0203] Further optionally, the step of dynamically updating the tabu length and capacity of the tabu list according to the filling situation of the tabu list and the frequency of error patterns includes:

[0204] Set an optimal interval for the tabu length, and randomly select a tabu length from the optimal interval for the tabu length as the initial tabu length;

[0205] If the optimal solution formed by combining the optimal candidate token in the current candidate sequence with the currently generated code sequence is superior to the currently recorded optimal solution in terms of the evaluation criterion, increase the taboo length;

[0206] Otherwise, decrease the taboo length;

[0207] Dynamically adjust the capacity of the taboo table according to the frequency of error patterns to ensure that the taboo table can effectively store and update error patterns.

[0208] Understandably, the taboo table can be used to store and avoid repeated access to those token patterns (or called bad candidate patterns) that are known to be non-optimal or will cause the generation of error codes during the decoding process. To make more efficient use of the taboo table, the taboo length and capacity of the taboo table can be dynamically adjusted according to its filling status and the occurrence frequency of error token patterns.

[0209] Specifically, first set an optimal interval for the taboo length and randomly select a value from this interval as the initial taboo length. This taboo length determines how long a certain error token pattern (or bad candidate pattern) will be retained in the taboo table, and thus affects the diversity and convergence speed of the generation process. For example, the initial taboo length of a dynamic language (such as Python) can be set to 5 - 8 steps. The initial taboo length of a static language (such as Java) can be set to 8 - 12 steps.

[0210] During the generation process, if the optimal candidate token (according to the probability value) in the current candidate sequence forms an optimal solution when combined with the currently generated code sequence, and this optimal solution is superior to the currently recorded optimal solution (i.e., the optimal solution found in the previous round or previous generation process), increase the taboo length. This allows this pattern (or a similar pattern) to be retained for a longer time in the subsequent code generation process, avoiding returning to sub-optimal token combinations prematurely, thereby promoting the generation process towards a more optimal token combination. On the contrary, if no solution containing a more optimal token combination is found, decrease the taboo length. This can increase the diversity of the generation, enabling the model to have the opportunity to explore token patterns (or bad candidate patterns) that were previously tabooed, and thus may discover new and more optimal token combinations.

[0211] In addition, the capacity of the taboo table can also be dynamically adjusted according to the frequency of error token patterns to ensure that the taboo table can store enough error token patterns without causing low storage and update efficiency due to excessive capacity. The capacity of the taboo table directly determines the number of error token patterns it can store.

[0212] In this embodiment, by dynamically adjusting the taboo length and capacity, the generation process becomes more efficient. This adjustment mechanism can not only avoid premature convergence to sub-optimal token combinations but also maintain the diversity of generation, thus better adapting to the characteristics of different code completion tasks and the generation stage.

[0213] S106, Dynamically select a sampling strategy according to the current decoding depth. Based on the selected sampling strategy, sample the next code snippet from the adjusted probability distribution, and finally generate a code completion suggestion that conforms to the abstract syntax tree rules and semantic constraints.

[0214] Dynamically select an appropriate sampling strategy according to the current decoding depth, sample the next code snippet from the probability distribution adjusted by the spherical probability mapping and taboo list mechanism, and finally generate a code completion suggestion that conforms to both the abstract syntax tree rules and semantic constraints. Its core goal is to establish a dynamic balance between syntax rules and generation freedom during the code completion process, improve the compilability and practicality of code completion, and at the same time balance the contradiction between exploration and exploitation to increase the compilation pass rate of long code snippets.

[0215] The steps of dynamically selecting a sampling strategy according to the current decoding depth, sampling the next code snippet from the adjusted probability distribution based on the selected sampling strategy, and finally generating a code completion suggestion that conforms to the abstract syntax tree rules and semantic constraints include:

[0216] During the code completion process, continuously monitor the length of the generated code sequence in real time to determine the current decoding depth;

[0217] When the current decoding depth is relatively shallow, adopt the Top-k sampling strategy, select the k code snippets with the highest probabilities in the probability distribution as candidates to enhance the diversity of the generated code completion suggestions;

[0218] When the current decoding depth is relatively deep, adopt the temperature sampling strategy, and control the randomness of the sampling process by adjusting the temperature parameter to ensure the semantic consistency of the generated code completion suggestions;

[0219] Based on the selected sampling strategy, sample the next code snippet from the adjusted probability distribution;

[0220] Add the sampled code snippet to the generated code sequence to form a code completion suggestion that conforms to the abstract syntax tree rules and semantic constraints.

[0221] Specifically, during the code completion process, the length of the generated code sequence is monitored in real time to determine the current decoding depth. When the current decoding depth is relatively shallow (e.g., <5 steps), the Top-k sampling strategy is adopted. This strategy selects the k code segments with the highest probabilities in the probability distribution as candidates. That is, at the initial stage of decoding, when the code generation is in the exploration phase, adopting this strategy can enhance the diversity of the generated code completion suggestions, encourage the model to generate more innovative code segments, and avoid getting stuck in local optima prematurely.

[0222] When the current decoding depth is relatively shallow (e.g., ≥5 steps), the temperature sampling strategy is adopted. The randomness of the sampling process is controlled by adjusting the temperature parameter. That is, in the later stage of decoding, more attention needs to be paid to semantic consistency in code generation to ensure that the generated code can be correctly compiled and run. The temperature sampling strategy can reduce randomness to a certain extent, making the model more inclined to select code segments with higher probabilities, thus ensuring the semantic consistency of the generated code completion suggestions.

[0223] Based on the selected sampling strategy, the next code segment is sampled from the probability distribution adjusted by the spherical probability mapping and the taboo list mechanism. Among them, the spherical probability mapping constrains the generated probability distribution within the effective probability subspace to ensure that the generated code is syntactically correct; the taboo list mechanism filters out syntax error candidates in real time, further improving the quality of the generated code.

[0224] The sampled code segment is added to the generated code sequence to form a code completion suggestion that conforms to the abstract syntax tree rules and semantic constraints. In this way, a complete code completion result is gradually constructed to ensure that the generated code meets both the syntax requirements and has reasonable semantics.

[0225] This embodiment ensures that the model can find a balance between exploration and exploitation at different decoding depths and generate high-quality code by dynamically selecting the sampling strategy.

[0226] The above-described embodiments only represent several implementation manners of the present invention. Their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the patent of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the appended claims.

Claims

1. A coding dynamic completion method for large language models, characterized in that The method includes: Constructing a mixed training dataset, which contains multilingual code snippets, syntax constraint annotations, and domain pre-trained corpora; Initializing a Transformer model architecture that integrates probability space topological transformation, and defining the hyper-sphere dimension and dynamic effective region parameters; Based on the mixed training dataset, performing model training with collaborative optimization of the main objective and auxiliary objectives, where the main objective is the code completion accuracy, and the auxiliary objectives include syntax tree validity prediction and semantic constraint satisfaction discrimination; At each decoding step, the model calculates the probability distribution of the next token based on the current context; In the model decoding stage, the generated probability distribution is constrained within the effective probability subspace through spherical probability mapping, and combined with the taboo table mechanism, syntax error candidates are filtered in real-time and the probability distribution is adjusted; Dynamically select a sampling strategy according to the current decoding depth. Based on the selected sampling strategy, sample the next code snippet from the adjusted probability distribution, and finally generate code completion suggestions that conform to the abstract syntax tree rules and semantic constraints; Among them, the steps of initializing the Transformer model architecture that integrates probability space topological transformation include: Inserting a probability space transformation module into the standard Transformer architecture; Defining hyper-sphere probability space parameters; Setting the dynamic effective region parameters and constraining the generated probability distribution through a spherical mapping function; Loading domain pre-trained weights when initializing the model parameters; The step of, at each decoding step, the model calculating the probability distribution of the next token according to the currently generated code sequence includes: Converting the currently generated code sequence into an embedding vector; Through the masked self-attention mechanism, capturing the internal dependencies of the currently generated code sequence; Taking the output of the self-attention layer as the query, and the output of the encoder as the key and value; Integrating the output of the encoder and the information of the currently generated code sequence, performing a non-linear transformation on the integrated information to generate the final output representation; Mapping the final output representation to a space of the vocabulary size through a linear layer; Using the Softmax function to generate the prediction probability distribution of each candidate token in the output sequence as the next token, and taking the set of all candidate tokens in the output sequence as the candidate sequence; The step of constraining the generated probability distribution within the effective probability subspace through spherical probability mapping, and combining with the taboo table mechanism, filtering syntax error candidates in real-time and adjusting the probability distribution includes: Mapping the generated probability distribution to the unit sphere, and realizing uniform constraint of the probability distribution through octahedral parameterization; Creating a taboo table for recording the error patterns and their probability characteristics generated in the historical decoding process; At each decoding step, query the taboo table according to the current candidate sequence to filter known error patterns, where the candidate sequence is the sequence composed of all possible tokens generated by the model for the next token position; Recalculating the spherical probability for the filtered candidate sequence to increase the generation probability of valid candidate tokens; Dynamically adjusting the spherical mapping parameters and taboo table update strategy according to the decoding progress.

2. The encoding dynamic completion method for large language models according to claim 1, characterized in that, The steps of mapping the generated probability distribution to the unit sphere and realizing the uniform constraint of the probability distribution through octahedral parameterization include: Convert the high-dimensional vector representation of each generated candidate token into spherical coordinates through the octahedral mapping formula. The mapping formula is: |x| + |y| + |z| = 1, where x, y, and z are the coordinates of the generated candidate token in three-dimensional space respectively; Normalize the mapped coordinates to ensure that all candidate tokens are located on the unit sphere; Calculate the probability value of each candidate token through the spherical coordinate system so that the probability distribution satisfies the uniformity and validity of the spherical space.

3. The encoding dynamic completion method for large language models according to claim 1, wherein The steps of querying the taboo table according to the current candidate sequence and filtering known error patterns at each decoding step include: Perform a syntax check on the currently generated candidate sequence and extract error patterns; Query the taboo table. If the candidate sequence contains the error patterns in the taboo table, filter it out; Update the filtered candidate sequence.

4. The encoding dynamic completion method for large language models according to claim 1, wherein The steps of recalculating the spherical probability of the filtered candidate sequence to increase the generation probability of valid candidate tokens include: Resample the filtered candidate sequence through the spherical probability mapping method; Calculate the spherical probability value of each valid candidate token and adjust the probability distribution according to the spherical probability value so that the adjusted probability distribution is more concentrated on valid candidates.

5. The encoding dynamic completion method for large language models according to claim 1, characterized in that The steps of dynamically adjusting the spherical mapping parameters and the taboo table update strategy according to the decoding progress include: Monitor the code generation quality and syntax validity during the decoding process and adjust the spherical mapping parameters in real time; Dynamically update the taboo length and capacity of the taboo table according to the filling situation of the taboo table and the frequency of error patterns; Through the reinforcement learning mechanism, give higher taboo priorities to the candidates that lead to error patterns and optimize the self-update strategy of the taboo table.

6. The encoding dynamic completion method for large language models according to claim 5, wherein The steps of dynamically updating the taboo length and capacity of the taboo table according to the filling situation of the taboo table and the frequency of error patterns include: Set the optimal interval of the taboo length and randomly select a taboo length from the optimal interval of the taboo length as the initial taboo length; If the optimal solution formed by combining the optimal candidate token in the current candidate sequence with the currently generated code sequence is better than the currently recorded optimal solution in terms of the evaluation criteria, increase the taboo length; Otherwise, decrease the taboo length; Dynamically adjust the capacity of the taboo table according to the frequency of error patterns to ensure that the taboo table can effectively store and update error patterns.

7. The encoding dynamic completion method for large language models according to claim 1, wherein The steps of dynamically selecting a sampling strategy according to the current decoding depth, sampling the next code segment from the adjusted probability distribution based on the selected sampling strategy, and finally generating a code completion suggestion that conforms to the abstract syntax tree rules and semantic constraints include: During the code completion process, monitor the length of the generated code sequence in real time to judge the current decoding depth; When the current decoding depth is relatively shallow, adopt the Top-k sampling strategy and select the k code segments with the highest probability in the probability distribution as candidates; When the current decoding depth is relatively deep, adopt the temperature sampling strategy and control the randomness of the sampling process by adjusting the temperature parameter. Sample the next code snippet from the adjusted probability distribution based on the selected sampling strategy; Add the sampled code snippet to the generated code sequence to form a code completion suggestion that conforms to the abstract syntax tree rules and semantic constraints.

Images