A large model training method, system, device and medium for low-code development

Abstract

The application discloses a large model training method and system for low-code development, equipment and medium, mainly relates to the technical field of large model training, in order to solve the problem that the existing scheme cannot effectively capture the syntax level relationship and block structure in the code, when processing the code with complex scope, it is easy to cause variable misreference and scope error, the existing fine tuning method ignores the pretraining knowledge of the model, resulting in unstable training process and slow convergence speed. Including: taking the input data of token sequence, the abstract syntax tree corresponding to the output code and the hierarchical code sequence as sample data, and constructing a training set; performing data enhancement operation on the sample code corresponding to the sample data in the training set; unfreezing the self-attention layer parameters of the encoder, setting the alignment loss function of the input data and the code sequence as the objective function; after completing the self-attention layer parameter adjustment, unfreezing all parameters of the decoder, and using the adversarial loss to adjust the large model.

Description

Technical Field

[0001] This application relates to the field of large model training technology, and in particular to a method, system, device and medium for training large models for low-code development. Background Technology

[0002] Existing code generation methods face numerous challenges, particularly in handling complex syntactic structures, variable scope, and high-level dependencies. While large pre-trained language models like GPT and BERT have demonstrated powerful capabilities in processing natural language text, they still present many problems in code generation tasks, especially in low-code development scenarios.

[0003] Specifically, this includes: 1. Traditional code generation methods treat code as a linear sequence, ignoring the structured hierarchical relationships within the code. This leads to the model's inability to accurately capture nested and dependent code blocks, resulting in syntax errors. 2. The Transformer's global attention mechanism cannot effectively limit the model's focus on scope, easily causing incorrect cross-scope references. When dealing with code with complex scope structures, variable misinterpretation is prone to occur. 3. Conventional fine-tuning methods often directly fine-tune the entire model, disrupting the general language representation learned during pre-training. Furthermore, the training process is not stable enough, making it difficult for existing methods to optimize specific tasks without losing prior knowledge. 4. In traditional code generation, the model often overemphasizes already generated parts, leading to the repeated generation of similar code snippets and increasing the proportion of redundant code. 5. Existing technologies struggle to handle variable conflicts within scopes, especially in complex function nesting and closure scenarios. The model is prone to misinterpreting variables with the same name, resulting in incorrect variable references when generating code. Summary of the Invention

[0004] This application provides a method, system, device, and medium for training large models for low-code development, in order to solve the above-mentioned problems.

[0005] Firstly, this application provides a method for training large models for low-code development, the method comprising:

[0006] The input data is processed into a token sequence; the output code is parsed into an abstract syntax tree, which in turn generates a hierarchical code sequence of the output code; the input data of the token sequence, the abstract syntax tree corresponding to the output code, and the hierarchical code sequence are used as sample data to construct a training set; data augmentation operations are performed on the sample code corresponding to the sample data in the training set to update the training set;

[0007] Based on the hierarchical code sequence, a dynamic hierarchical mask matrix for the large model is generated to restrict each token to only focus on tokens in the current level or adjacent parent / child levels.

[0008] Freeze the position encoding matrix and word embedding matrix of the large model; divide the attention head of the decoder into two groups: syntax head and semantic head, and configure the corresponding preset constraints;

[0009] Unfreeze the self-attention layer parameters of the encoder, freeze the decoder parameters, and set the alignment loss function between the input data and the code sequence as the objective function. After adjusting the self-attention layer parameters, unfreeze all decoder parameters and apply adversarial loss to adjust all decoder parameters. Dynamically adjust the pruning threshold based on gradient direction similarity. Add a scope encoding vector to the decoder embedding layer to achieve explicit encoding of scope hierarchy. Apply a coverage penalty term to the decoder attention to adjust the overall model. Obtain the adjusted overall model.

[0010] In one implementation of this application, the output code is parsed into an abstract syntax tree, thereby generating a hierarchical code sequence of the output code, specifically including:

[0011] The output code is parsed into an abstract syntax tree, and a hierarchical code sequence with hierarchical tags is generated by depth-first traversal. The hierarchy of each code block in the hierarchical code sequence is marked, so that the large model can perceive the code structure through the delimiter tags.

[0012] In one implementation of this application, data augmentation is performed on the sample codes corresponding to the sample data in the training set, specifically including:

[0013] Parse the abstract syntax tree corresponding to the sample code and identify preset identifiers; the preset identifiers include at least: user-defined variables, function names, and class names; generate random alternative names that conform to preset rules for each preset identifier;

[0014] Using hash mapping or random strings ensures that references within the same scope are synchronously replaced and updated.

[0015] Perform syntax structure pattern matching on the abstract syntax tree, and replace the subtrees that successfully match the abstract syntax tree with the corresponding preset new code.

[0016] In one implementation of this application, the attention header of the decoder is divided into two groups: a syntax header and a semantic header, and corresponding preset constraints are configured, specifically including:

[0017] The default constraints corresponding to the syntax header are defined by the following formula:

[0018] Calculate the syntax head attention result ;

[0019] in, Represents the query matrix. Represents the key matrix. Represents a value matrix, This represents the Softmax function. This represents the attention head dimension. Represents the attention mask matrix, for transpose;

[0020] The preset constraints corresponding to the semantic header are defined by the following formula:

[0021] Calculate the semantic head attention calculation result ;

[0022] in, Represents the intra-block mask matrix. This indicates element-wise multiplication.

[0023] In one implementation of this application, the self-attention layer parameters of the encoder are unfrozen, the decoder parameters are frozen, and the alignment loss function between the input data and the code sequence is set as the objective function, specifically including:

[0024] Obtain the hierarchical consistency loss and syntax keyword matching loss;

[0025] Based on the hierarchical consistency loss and syntax keyword matching loss, the code error constraint loss is calculated.

[0026] By utilizing code error constraint loss and the hidden state representation of the input data after encoder processing, the alignment loss function between the input data and the code sequence is calculated.

[0027] In one implementation of this application, adversarial loss is used to adjust all parameters of the decoder, specifically including:

[0028] Obtain cross-entropy loss and adversarial loss;

[0029] The total loss function is obtained by weighted summation of cross-entropy loss and adversarial loss; in the adversarial loss network, a syntax validator is used as the discriminator.

[0030] In one implementation of this application, the pruning threshold is dynamically adjusted based on gradient direction similarity, specifically including:

[0031] Through the formula: Calculate the clipping threshold ;

[0032] in, For the first The gradient vector at each time step. For the first The gradient vector at each time step. It is the L2 norm. This is the scaling factor; for The transpose of .

[0033] Secondly, this application provides a large model training system for low-code development, the system comprising:

[0034] The data processing module is used to process the input data into a token sequence; parse the output code into an abstract syntax tree, and then generate a hierarchical code sequence of the output code; use the input data of the token sequence, the abstract syntax tree corresponding to the output code, and the hierarchical code sequence as sample data to construct a training set; and perform data augmentation operations on the sample code corresponding to the sample data in the training set to update the training set.

[0035] The model training module generates a dynamic hierarchical mask matrix for a large model based on hierarchical code sequences, restricting each token to focus only on tokens in the current level or adjacent parent / child levels. It freezes the positional encoding matrix and word embedding matrix of the large model; divides the decoder's attention head into two groups: a syntax head and a semantic head, and configures corresponding preset constraints; unfreezes the encoder's self-attention layer parameters, freezes the decoder parameters, and sets the alignment loss function between the input data and the code sequence as the objective function; after adjusting the self-attention layer parameters, unfreezes all decoder parameters and applies adversarial loss to adjust all decoder parameters; dynamically adjusts the pruning threshold based on gradient direction similarity; adds a scope encoding vector to the decoder embedding layer to explicitly encode the scope hierarchy; and applies a coverage penalty term to the decoder attention to adjust the large model; finally, it obtains the adjusted large model.

[0036] Thirdly, this application provides a large model training device for low-code development, the device comprising:

[0037] processor;

[0038] And a memory that stores executable code, which, when executed, causes the processor to perform a large model training method for low-code development, as described above.

[0039] Fourthly, this application provides a non-volatile computer storage medium storing computer instructions, which, when executed, implement a large model training method for low-code development as described above.

[0040] As can be seen from the above technical solutions, this application has the following advantages:

[0041] 1. Traditional code generation methods typically process code into flat text, making it difficult for models to understand the nested structure and syntactic dependencies of the code. This application significantly improves the model's understanding of the nested levels of code blocks and reduces the syntax error rate by converting the code into an abstract syntax tree and using depth-first traversal to generate a linear sequence with hierarchical tags.

[0042] 2. Traditional code generation models usually rely on fixed code structures and are difficult to adapt to different forms of code expression. This application proposes a data augmentation method based on abstract syntax trees for code variable renaming and equivalent syntax structure replacement. By renaming variables and replacing syntax structures, the generalization ability of the model is enhanced, and the robustness and flexibility of the model are improved while avoiding excessive reliance on specific variable names or syntax structures.

[0043] 3. Existing Transformer models use a global attention mechanism, which can easily overlook the scope limitations of variables in the code, leading to incorrect cross-scope references. This application constructs a dynamic hierarchical attention mask matrix to ensure that each token can only pay attention to tokens in the current level or adjacent parent / child levels, thereby strictly adhering to the syntax scope rules during the generation process and effectively reducing incorrect variable references and cross-scope information interference.

[0044] 4. Existing fine-tuning methods are usually full-model fine-tuning, which can easily lead to the loss of pre-training knowledge or model training instability. The phased progressive fine-tuning strategy proposed in this application first fine-tunes the encoder part and then gradually unfreezes the decoder part, ensuring that the model can gradually optimize code generation capabilities without losing pre-training knowledge, thereby improving training stability and convergence speed.

[0045] 5. In traditional multi-head attention mechanisms, all attention heads are equal and cannot distinguish between different needs of syntactic structure and semantics. This application divides the attention heads of the decoder into two groups, syntactic head and semantic head, and optimizes them separately, so that the model can pay attention to the syntactic structure and semantic relationship of the code at the same time, thereby improving the accuracy and syntactic consistency of code generation.

[0046] 6. The code generated by the traditional Transformer model is prone to redundancy, especially when processing code, it may repeatedly generate code blocks with similar structures. This application avoids the decoder repeatedly focusing on the generated parts by adopting a coverage penalty mechanism, thereby reducing the generation of redundant code and improving the efficiency of code generation.

[0047] 7. In traditional code generation methods, it is impossible to distinguish the meaning of variables with the same name in different scopes, which may lead to misparsing problems. This application adds explicit scope encoding to the decoder embedding layer, so that the model can accurately distinguish the meaning of variables with the same name in different scopes, effectively avoiding scope-related parsing errors. Attached Figure Description

[0048] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart of a large model training method for low-code development provided in an embodiment of this application.

[0050] Figure 2 This is a schematic diagram showing the error rate comparison of different encoding methods with varying AST depth, provided in an embodiment of this application.

[0051] Figure 3 This is a schematic diagram of variable scope error type analysis provided in an embodiment of this application.

[0052] Figure 4 This is a schematic diagram comparing training loss dynamics curves provided in an embodiment of this application.

[0053] Figure 5 This is a schematic diagram illustrating the variable resolution performance in a complex scope scenario provided in an embodiment of this application.

[0054] Figure 6 This is a schematic diagram of the internal structure of a large model training system for low-code development provided in an embodiment of this application.

[0055] Figure 7 This is a schematic diagram of the internal structure of a large model training device for low-code development provided in an embodiment of this application. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] Those skilled in the art should understand that the embodiments described below are merely preferred embodiments of this disclosure and do not imply that this disclosure can only be implemented through these preferred embodiments. These preferred embodiments are merely used to explain the technical principles of this disclosure and are not intended to limit the scope of protection of this disclosure. Based on the preferred embodiments provided by this disclosure, all other embodiments obtained by those skilled in the art without creative effort should still fall within the scope of protection of this disclosure.

[0058] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0059] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0060] The embodiment provides a method for training large models for low-code development, such as Figure 1 As shown in the embodiments of this application, the method mainly includes the following steps:

[0061] Step 110: Process the input data into a token sequence; parse the output code into an abstract syntax tree, and then generate a hierarchical code sequence of the output code; use the input data of the token sequence, the abstract syntax tree corresponding to the output code, and the hierarchical code sequence as sample data to construct a training set; perform data augmentation operations on the sample code corresponding to the sample data in the training set to update the training set.

[0062] In some embodiments, the output code is parsed into an abstract syntax tree, thereby generating a hierarchical code sequence of the output code, specifically including:

[0063] The output code is parsed into an abstract syntax tree, and a hierarchical code sequence with hierarchical tags is generated by depth-first traversal. The hierarchy of each code block in the hierarchical code sequence is marked, so that the large model can perceive the code structure through the delimiter tags.

[0064] It should be noted that the input data for the large model is a token sequence described in natural language, such as "create a Python function to calculate the Fibonacci sequence", which is processed into a token sequence by a tokenizer and marked with [CLS] / [SEP].

[0065] like, .

[0066] In addition, in the steps, the output code is parsed into an abstract syntax tree, and then a hierarchical code sequence of the output code is generated. This can be further specified as follows: the output data of the large model is a code sequence with hierarchical labels, such as: def fib(n): ifn <= 1: return n else: return fib(n-1) + fib(n-2);

[0067] The input text and output code data have strong structured characteristics, such as nested code blocks and syntax tree dependencies. Conventional methods treat the code directly as flat text, making it difficult for the model to capture the syntactic hierarchy.

[0068] This application parses the output code into an abstract syntax tree, and then generates a linear sequence with hierarchical tags through depth-first traversal, marking the hierarchy of each code block. This allows the model to perceive the code structure through delimiters, solving the problem of high syntax error rates caused by the flattening of conventional methods. The hierarchical sequence generation function is defined as follows:

[0069] ;

[0070] In the formula, To output a hierarchical sequence of code, representing the code sequence with hierarchical tags converted through an abstract syntax tree. For example, the code deff(): if x: return 1 is parsed into the generated sequence of the abstract syntax tree: [BLK_0]deff():[BLK_1]if x:[BLK_2]return 1;

[0071] This is the root node of the code, such as using a Python function definition node as the root.

[0072] This is a depth-first traversal function that recursively traverses the nodes of the AST, concatenates the tokens in depth-first order, and converts them into a linear sequence.

[0073] The step size controls the recursive step size, affecting the traversal order and depth, for example, ;

[0074] This is a hierarchy separator, indicating the nesting level of the current code block. This represents the depth of the current abstract syntax tree node, used to explicitly encode the nesting level of code blocks. For example, [BLK_0] represents the root node code block, and [BLK_1] represents a code block with a depth of 1, such as a function body.

[0075] This is a sequence concatenation operator.

[0076] This includes performing data augmentation on the sample codes corresponding to the sample data in the training set, specifically including:

[0077] Parse the abstract syntax tree corresponding to the sample code and identify preset identifiers; the preset identifiers include at least: user-defined variables, function names, and class names; generate random alternative names that conform to preset rules for each preset identifier;

[0078] Using hash mapping or random strings ensures that references within the same scope are synchronously replaced and updated.

[0079] Perform syntax structure pattern matching on the abstract syntax tree, and replace the subtrees that successfully match the abstract syntax tree with the corresponding preset new code.

[0080] More specifically, in order to adapt to the diversity of code surface forms, force the model to focus on code logic rather than variable naming or syntactic structure, and improve generalization ability, this application performs two types of enhancement operations on the code samples in the training set: code variable renaming and equivalent syntactic structure replacement, while keeping the functional semantics unchanged.

[0081] 1) Renaming code variables:

[0082] Parse the abstract syntax tree of the code, identify all user-defined variables, function names, class names and other identifiers, and generate random but legal alternative names for each identifier;

[0083] Use hash mapping or random strings to ensure that references within the same scope are updated synchronously. For example, renaming the function definition `def fib(n):...` to `def fibonacci(num):...` will synchronously replace all internal references.

[0084] Scope analysis using an abstract syntax tree ensures that renaming does not cross scope boundaries. For example, it still maintains the isolation between variables inside functions and global variables. At the same time, a symbol table is used to record the mapping relationship between the original identifier and the replaced identifier to avoid conflicts.

[0085] 2) Equivalent syntax structure substitution:

[0086] Define a syntax replacement rule base to cover common equivalent structures, such as loop transformations, conditional abbreviations, and API replacements;

[0087] Pattern matching is performed on the code abstract syntax tree, rules are applied to replace subtrees and generate new code, such as replacing `for i in range(n): print(i)` with `i=0`; `while i...` <n: print(i); i+=1。

[0088] By using static analysis to verify the functional equivalence of the replaced code, the replacement depth is limited, and unconventional code is avoided.

[0089] Step 120: Based on the hierarchical code sequence, generate a dynamic hierarchical mask matrix for the large model to restrict each token to only focus on tokens in the current level or adjacent parent / child levels.

[0090] It should be noted that the syntax rules of the code require that tokens within a specific scope can only focus on the local context. For example, variables inside functions in the code cannot reference external variables. The global attention mechanism of the conventional Transformer tends to ignore this constraint.

[0091] This application constructs a dynamic attention mask matrix to ensure that each token can only focus on tokens in the current level or adjacent parent / child levels. Specifically, a dynamic hierarchical mask matrix is ​​generated based on a hierarchical sequence. The elements of the attention mask matrix restrict each token to only focus on tokens to its left in the same level or adjacent parent / child levels, forcing the model to follow code syntax scoping rules and solving the problem of variable misreferences. This is represented as follows:

[0092] ;

[0093] In the formula, These are elements of the attention mask matrix, indicating whether the i-th token can pay attention to the j-th token. They are used to control the attention scope of the model and ensure that the model follows the scope rules in the code.

[0094] The depth of the hierarchy label at position i, for example, [BLK_2] corresponds to ;

[0095] Label the depth of the hierarchy at position j;

[0096] To limit the maximum allowed depth difference, the model focuses on a specific context. By restricting the hierarchical gap, the model only pays attention to tokens in adjacent levels or parent / child levels, avoiding cross-level information interference. .

[0097] Step 130: Freeze the positional encoding matrix and word embedding matrix of the large model; divide the decoder's attention head into two groups: a syntax head and a semantic head, and configure the corresponding preset constraints; unfreeze the encoder's self-attention layer parameters, freeze the decoder parameters, and set the alignment loss function between the input data and the code sequence as the objective function; after completing the self-attention layer parameter adjustment, unfreeze all decoder parameters and use adversarial loss to adjust all decoder parameters; dynamically adjust the pruning threshold based on gradient direction similarity; add a scope encoding vector to the decoder embedding layer to achieve explicit encoding of the scope hierarchy; and use a coverage penalty term in the decoder attention to adjust the large model; obtain the adjusted large model.

[0098] It should be noted that since pre-trained large models have already learned general semantic representations and basic code patterns on massive natural language and code corpora, training from scratch requires huge computing power and is difficult to converge to the same level. Fine-tuning can take advantage of the pre-trained model's ability to understand natural language intent and generate basic code, and adapt to specific code generation tasks by adjusting a small number of parameters, which can significantly reduce training costs and improve convergence speed.

[0099] This application freezes some parameters of the pre-trained model while retraining the large model, i.e. fine-tuning, and improves the quality of code generation through structured data processing and targeted optimization of the encoder-decoder layer.

[0100] The encoder-decoder layer is located inside the stacked Transformer layers and is responsible for mapping the semantic features of the input text to the syntactic structure of the target code. For example, in a 24-layer Transformer, layers 4-20 are fine-tunable encoder-decoder layers. Freezing other layers (including word embedding layers and positional encoding layers) can preserve the language prior knowledge obtained from pre-training, and only optimize the intermediate layers to focus on code structure modeling. Since the fine-tuned large model is used for code generation, that is, input text description, output code.

[0101] In this step, the position encoding matrix and word embedding matrix of the large model are frozen, which can be specifically:

[0102] Freeze the position encoding matrix and word embedding matrix, and only update the self-attention and feedforward network parameters of the encoder and decoder;

[0103] At the same time, the position encoding matrix is ​​defined as The word embedding matrix is , This represents the maximum sequence length.

[0104] Understandably, the word embedding layers and positional encodings of pre-trained large models are usually designed for natural language, and direct fine-tuning can easily destroy their semantic priors.

[0105] In some embodiments, the attention header of the decoder is divided into two groups: a syntax header and a semantic header, and corresponding preset constraints are configured, specifically including:

[0106] The default constraints corresponding to the syntax header are defined by the following formula:

[0107] Calculate the syntax head attention result ;

[0108] in, Represents the query matrix. Represents the key matrix. Represents a value matrix, This represents the Softmax function. This represents the attention head dimension. Represents the attention mask matrix,

[0109] The preset constraints corresponding to the semantic header are defined by the following formula:

[0110] Calculate the semantic head attention calculation result ;

[0111] in, Represents the intra-block mask matrix. This indicates element-wise multiplication.

[0112] Dividing the decoder's attention head into two groups—a syntax head and a semantic head—and configuring corresponding preset constraints can be further specified as follows:

[0113] When generating code, some attention heads need to focus on syntactic structures, such as bracket matching and indentation levels, while others need to understand semantic relationships such as variable naming and function calls. In the multi-head attention mechanism of the decoder, conventional multi-head attention treats all heads equally and cannot distinguish between syntactic structures and ordinary semantic relationships.

[0114] This application divides the attention head of the decoder into two groups: a syntax head and a semantic head, and applies different constraints to each group, so that the model can capture both syntax rules and local semantics at the same time, thus solving the problem that a single attention mechanism cannot take into account both structure and semantics.

[0115] 1) The syntax header focuses on syntax structure, such as bracket matching and indentation levels. It modifies attention weights using an attention mask matrix. The syntax header calculates the modified attention weights using the attention mask matrix, ensuring that it focuses on the syntax structure of the code, such as bracket matching and indentation levels. This is represented as:

[0116] ;

[0117] In the formula, Here is the attention mask matrix, whose elements are: ;

[0118] This represents the logarithmic function, with a default base of 10.

[0119] This indicates the result of the syntax head attention calculation;

[0120] Represents the query matrix;

[0121] Represents the key matrix;

[0122] Represents a value matrix;

[0123] For the Softmax function;

[0124] for transpose;

[0125] The attention head dimension is typically... , The number of heads.

[0126] 2) The semantic head focuses on the semantic relationships between variables, functions, etc. in the code, maintaining the global attention mechanism. The semantic head retains the original global attention but limits its scope to only cover the current code block, enabling the model to recognize local semantics within different code blocks, as shown below:

[0127] ;

[0128] In the formula, The block mask matrix has the following elements: ;

[0129] For elements of the intra-block mask matrix, It means if and only if the first The position and the first These positions belong to the same code block, through Mark the judgment, if the first The position and the first If there are no code block markers of different depths between locations, they are considered to be the same block; otherwise... ;

[0130] This represents the result of semantic head attention calculation;

[0131] This is an element-wise multiplication.

[0132] In this step, the self-attention layer parameters of the encoder are unfrozen, the decoder parameters are frozen, and the alignment loss function between the input data and the code sequence is set as the objective function. After adjusting the self-attention layer parameters, all decoder parameters are unfrozen, and adversarial loss is applied to adjust all decoder parameters. Specifically:

[0133] Directly fine-tuning all trainable parameters can easily cause the model to forget pre-trained knowledge;

[0134] This application adopts a two-stage unfreezing strategy, which first enhances the semantic understanding ability of the encoder and then optimizes the code generation ability of the decoder, thereby solving the problem of insufficient optimization caused by parameter coupling in end-to-end fine-tuning.

[0135] 1) Stage 1: Only unfreeze the self-attention layer parameters of the encoder, fix the decoder parameters, and use the alignment loss between the input text and the code sequence as the objective function. The loss function is:

[0136] ;

[0137] In the formula, To align the loss, the encoder output is forced to match the decoder's hidden state;

[0138] The length of the target sequence;

[0139] Indicates the given input and historical output Time prediction The probability of;

[0140] The hidden state representation of the input natural language description after processing by the encoder, for example, encoding "create a Python function to calculate the Fibonacci sequence" into a high-dimensional vector sequence;

[0141] For the first in the target sequence For example, when generating a code sequence, if the current generated sequence is [BLK_2]return n, then... It could be a return;

[0142] Loss due to code error constraints;

[0143] Indicates the first [number]th [item] in the target sequence All tokens generated up to position [BLK_2] return n It contains [BLK_0]def fib(n): [BLK_1]if n<=1.

[0144] Furthermore, the code error constraint loss employs a dual mechanism of hierarchical consistency verification and immediate syntax validation to explicitly constrain the model to generate code sequences that conform to grammatical rules. The code error constraint loss consists of two parts:

[0145] ;

[0146] In the formula, This is a loss of hierarchical consistency, used to force the generated code to follow the rules of depth-first traversal, ensuring that code block nesting is legal;

[0147] The loss is the syntax keyword matching loss, used to ensure that keywords in the generated code conform to the syntax rules.

[0148] 1) Loss of consistency in hierarchical structure: Force the generated hierarchical tag sequence to follow the depth-first traversal rule to ensure that code block nesting is legal;

[0149] 2) Syntax Keyword Matching Loss: Constrain the position of key syntactic elements (such as colons, parentheses, keywords) to conform to the language specification.

[0150] Specifically, the hierarchical consistency loss is achieved by verifying the depth difference between adjacent hierarchical labels, defining the first... The hierarchical labeling depth at each location is The depth of the hierarchical marker at the previous position is The legal depth difference must meet the following requirements. A second penalty is applied to illegal depth differences, expressed as:

[0151] ;

[0152] In the formula, To adjust the constraint strength as a structural loss weight, for example, ;

[0153] This represents the total number of all hierarchical tags in the batch.

[0154] This indicates the legal depth difference. The loss is zeroed out, and penalties are only imposed for differences exceeding 1.

[0155] Furthermore, the syntax keyword matching loss is achieved by checking the contextual validity of key syntax elements, defining a keyword set. Keywords such as def, if, etc., if the model generates keywords in the context However, if it does not conform to the grammatical rules, a probability-weighted penalty is applied. A stronger penalty is imposed on high-probability generation behaviors at incorrect positions to suppress grammatical violations, as shown below:

[0156] ;

[0157] In the formula, Indicates the first in the context One location;

[0158] Loss weight for keywords, such as, ;

[0159] For indicator functions, when The value is 1 if the context conforms to the syntax rules, and 0 otherwise.

[0160] 2) Phase 2: Unfreeze all decoder parameters and apply adversarial loss;

[0161] Conventional cross-entropy loss only optimizes token-level predictions and cannot guarantee the syntactic correctness of the generated code;

[0162] This application employs a syntax validator as the discriminator. By constructing an adversarial loss, the model explicitly follows grammatical constraints during optimization, avoiding the generation of grammatically incorrect code. This solves the problem that traditional loss functions cannot directly encode grammatical rules, and is expressed as follows:

[0163] ;

[0164] In the formula, The total loss function is obtained by weighted summation of cross-entropy loss and adversarial loss;

[0165] Cross-entropy loss;

[0166] These are the balance coefficients of the loss function;

[0167] This indicates sampling code from the code distribution generated by the generator, calculating the expected log probability of the sampled code after passing the discriminator. For example, if the generated code is verified to have no syntax errors using ast.parse(), this would be considered a valid sample. ;

[0168] The generator and discriminator are two modules of a generative adversarial network;

[0169] The code distribution generated by the generator;

[0170] The result of the code passing static syntax checking is verified by an abstract syntax tree parser, such as Python's `ast.parse()`. If no errors are found, it returns 1. This indicates that there are no syntax errors.

[0171] In this step, the cropping threshold is dynamically adjusted based on gradient direction similarity, specifically including:

[0172] This application dynamically adjusts the clipping threshold based on gradient direction similarity, alleviating the inadequacy of traditional clipping under anisotropic parameter space and improving training stability, as expressed in:

[0173] ;

[0174] In the formula, For the first The gradient vector at each time step, such as the gradient of the encoder's self-attention layer;

[0175] For the first The gradient vector at each time step is used to calculate directional similarity.

[0176] for transpose;

[0177] The pruning threshold is set as follows: when the gradient direction is consistent with the historical average direction, such as both pointing in the direction of loss decrease, the threshold is increased; otherwise, it is decreased. By dynamically adjusting the pruning threshold based on the similarity of the gradient direction, the fixed threshold pruning is avoided from disrupting the update of deep parameters.

[0178] It is an L2 norm;

[0179] For scaling factors, such as, .

[0180] The step axis involves adding a scoped encoding vector to the decoder embedding layer to achieve explicit encoding of the scope hierarchy. Specifically, this can be done as follows:

[0181] The scopes of variables in the code are nested, and variables with the same name in different scopes have different meanings. Conventional positional encoding cannot distinguish the meaning of variables with the same name in different scopes.

[0182] This application adds a scope encoding vector to the decoder embedding layer. By explicitly encoding the scope hierarchy, the model can distinguish between variables with the same name in different contexts, thus solving the problem of variable misinterpretation. This is represented as:

[0183] ;

[0184] In the formula, The first in the current scope stack Layer identifiers, such as function names, class names, etc.;

[0185] It is a two-layer fully connected network with the GELU activation function, which maps the scope information into a vector with the same dimension as the word embedding;

[0186] The current scope stack depth, such as the function nesting level;

[0187] To integrate scope information, word embedding incorporates scope information to ensure that variables with the same name do not conflict in different scopes;

[0188] Embedding of original words;

[0189] For position encoding;

[0190] A coverage penalty term is used in the decoder attention to adjust the overall model. Specifically, this can be done as follows:

[0191] Code generation often results in the repeated generation of the same structure, such as redundant parentheses, which is caused by the attention mechanism's excessive focus on already generated fragments.

[0192] This application employs a coverage penalty term in the decoder attention to suppress repeated attention to already generated parts and reduce redundant code generation, expressed as:

[0193] ;

[0194] In the formula, For the first The position is the first The attention weight at each position represents the influence of the tokens that the decoder pays attention to during the generation process on the current token. For example, the attention weight at the 3rd position when decoding the 5th token.

[0195] For parameter update operations;

[0196] The penalty intensity represents the cumulative penalty for each historical step's attention to the same position, inhibiting repeated generation, such as... ;

[0197] In the first At time step, the decoder performs a step on the first... Attention weights for each position, and, From the first time step to the second Between time steps, for example, when generating the 5th token, if the historical steps Position attention weights Then the current step of The penalty will decrease due to cumulative punishment.

[0198] After obtaining the trained large model, this application also includes:

[0199] After fine-tuning the large model based on the Transformer model architecture, the inference process mainly includes inputting natural language descriptions and generating corresponding code, as follows:

[0200] First, the user inputs a token sequence described in natural language (e.g., "create a Python function to calculate the Fibonacci sequence"). This input is converted into a token format that the model can process by a token segmenter, and then processed by the model's encoder. The encoder part transforms the input natural language description into a high-dimensional semantic representation while preserving its contextual information.

[0201] Then, the encoder's output is further processed by the decoder, which generates code step by step based on the context information provided by the encoder and the currently generated part.

[0202] During the generation process, the attention mechanism used by the decoder dynamically adjusts the scope of the attention context according to different task requirements, including attention to syntax structure, variables, function names, etc.

[0203] To avoid generating duplicate or erroneous code, the attention mechanism in the decoder incorporates a coverage penalty to reduce redundant generation.

[0204] When the decoder finishes generating the code, the output layer generates a probability distribution through the Softmax layer, selects the most likely token as the next code element, and continues until the entire code sequence is generated.

[0205] Finally, the generated code will be returned according to the task requirements. Users can use the code directly in different programming environments, or further optimize and deploy it by integrating it into a low-code development platform.

[0206] like Figure 2As shown, to verify the ability of the hierarchical coding method to model nested code structures, the changes in syntax error rate of traditional flat coding and the abstract syntax tree hierarchical coding proposed in this application at different nesting depths were compared. The experimental results show that as the nesting level of code blocks deepens, the error rate of the traditional method increases rapidly, while the hierarchical coding method maintains a low error level. The filled area between the two curves in the figure intuitively shows the performance difference between the two methods in deep nesting scenarios. The flat curve of hierarchical coding benefits from the explicit hierarchical separation mark, which enables the model to accurately identify the parent-child relationship of code blocks and effectively avoid typical problems such as mismatched brackets and indentation errors caused by structural misunderstanding. It shows significant advantages in handling multiple nested loops or complex condition judgments.

[0207] like Figure 3 As shown, to analyze the improvement effect of dynamic hierarchical attention masking on variable scope resolution, a radar chart is used to compare the performance of global attention mechanism and hierarchical attention mechanism in four typical variable error scenarios. The difference in polygon area in the chart clearly shows that the traditional method has obvious defects in problems such as global variable misreference and local variable overwriting. Hierarchical attention, by constraining the attention flow between different levels of code blocks, significantly reduces the occurrence rate of various scope errors. Each dimension on the radial coordinate axis corresponds to a type of error. This method forms more compact inward polygons in all dimensions, proving that the dynamic mask matrix can accurately control the attention range, prevent cross-scope information interference, and thus significantly improve the accuracy of variable reference.

[0208] like Figure 4 As shown, the optimization efficiency of the progressive parameter unfreezing strategy is verified through dynamic analysis of the training loss curve. The two curves in the figure represent the training processes of end-to-end fine-tuning and phased unfreezing, respectively. The unit of the horizontal axis is "thousand iterations". The curve of this method shows a steeper downward trend after the phase switching point, indicating that the phased optimization strategy effectively alleviates the conflict problem of parameter update. The phase switching time marked by the vertical dashed line shows that the priority optimization of encoder parameters provides a stable semantic representation basis for decoder. The oscillation and fluctuation of traditional methods expose the optimization instability caused by parameter coupling. The smooth convergence state of the curve in the later stage proves that the phased unfreezing strategy achieves rapid convergence and stability of the training process through an ordered parameter activation mechanism.

[0209] like Figure 5As shown, to analyze the variable resolution performance in complex scope scenarios, an error bar chart is used to compare the performance of different methods in scenarios such as function nesting and closures. The data points with error bars in the chart show that as the complexity of the code scope increases, the accuracy of traditional methods decreases significantly and the fluctuation range expands, while our method consistently maintains high accuracy and low variance. Especially in closure functions and multi-threaded callback scenarios, the significant upward shift of the data points of our method verifies the effectiveness of scope stack encoding. By incorporating scope hierarchy information into word vector representation, the model can accurately track the binding relationship of variables and avoid ambiguous resolution of variables with the same name in different contexts. The difference in error bar length further indicates that explicit encoding enhances the robustness of the model to scope boundary determination.

[0210] In addition, this application Figure 6 This application provides a large model training system for low-code development. For example... Figure 6 As shown in the embodiments of this application, the system mainly includes:

[0211] The data processing module 210 is used to process the input data into a token sequence; parse the output code into an abstract syntax tree, and then generate a hierarchical code sequence of the output code; use the input data of the token sequence, the abstract syntax tree corresponding to the output code, and the hierarchical code sequence as sample data to construct a training set; and perform data augmentation operations on the sample code corresponding to the sample data in the training set to update the training set.

[0212] The model training module 220 is used to generate a dynamic hierarchical mask matrix for a large model based on a hierarchical code sequence, restricting each token to only focus on tokens in the current level or adjacent parent / child levels; freezing the positional encoding matrix and word embedding matrix of the large model; dividing the decoder's attention head into two groups, a syntax head and a semantic head, and configuring corresponding preset constraints; unfreezing the encoder's self-attention layer parameters, freezing the decoder parameters, and setting the alignment loss function between the input data and the code sequence as the objective function; after completing the self-attention layer parameter adjustment, unfreezing all decoder parameters and using adversarial loss to adjust all decoder parameters; dynamically adjusting the pruning threshold based on gradient direction similarity; adding a scope encoding vector to the decoder embedding layer to achieve explicit encoding of scope hierarchy; and using a coverage penalty term in the decoder attention to adjust the large model; obtaining the adjusted large model.

[0213] The above are method embodiments of this application. Based on the same inventive concept, embodiments of this application also provide a large model training device for low-code development. Figure 7As shown, the device includes: a processor; and a memory storing executable code thereon, which, when executed, causes the processor to perform a large model training method for low-code development as described in the above embodiments.

[0214] Specifically, the server processes the input data into a token sequence; parses the output code into an abstract syntax tree, and then generates a hierarchical code sequence of the output code; uses the input data of the token sequence, the abstract syntax tree corresponding to the output code, and the hierarchical code sequence as sample data to construct a training set; performs data augmentation operations on the sample code corresponding to the sample data in the training set to update the training set; based on the hierarchical code sequence, generates a dynamic hierarchical mask matrix for the large model to restrict each token to only focus on tokens in the current level or adjacent parent / child levels; and freezes the positional encoding matrix and word embedding matrix of the large model. The decoder's attention head is divided into two groups: a syntax head and a semantic head, and corresponding preset constraints are configured. The self-attention layer parameters of the encoder are unfrozen, the decoder parameters are frozen, and the alignment loss function between the input data and the code sequence is set as the objective function. After adjusting the self-attention layer parameters, all decoder parameters are unfrozen, and adversarial loss is used to adjust all decoder parameters. The pruning threshold is dynamically adjusted based on gradient direction similarity. A scope encoding vector is added to the decoder embedding layer to achieve explicit encoding of the scope hierarchy. A coverage penalty term is used in the decoder attention to adjust the overall model. The adjusted overall model is then obtained.

[0215] In addition, embodiments of this application also provide a non-volatile computer storage medium storing executable instructions, which, when executed, implement the large model training method for low-code development as described above.

[0216] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for training large models for low-code development, characterized in that, The method includes: The input data is processed into a token sequence; the output code is parsed into an abstract syntax tree, which in turn generates a hierarchical code sequence of the output code; the input data of the token sequence, the abstract syntax tree corresponding to the output code, and the hierarchical code sequence are used as sample data to construct a training set; data augmentation operations are performed on the sample code corresponding to the sample data in the training set to update the training set; Based on the hierarchical code sequence, a dynamic hierarchical mask matrix for the large model is generated to restrict each token to only focus on tokens in the current level or adjacent parent / child levels. Freeze the position encoding matrix and word embedding matrix of the large model; The decoder's attention head is divided into two groups: a syntax head and a semantic head, and corresponding preset constraints are configured; these include: The default constraints corresponding to the syntax header are defined by the following formula: Calculate the syntax head attention calculation result in, Represents the query matrix. Represents the key matrix. for transpose, Represents a value matrix, This represents the Softmax function. This represents the attention head dimension. Represents the attention mask matrix, The preset constraints corresponding to the semantic header are defined by the following formula: Calculate the semantic head attention calculation results ;in, Represents the intra-block mask matrix. Indicates element-wise multiplication; Unfreeze the self-attention layer parameters of the encoder, freeze the decoder parameters, and set the alignment loss function between the input data and the code sequence as the objective function. After adjusting the self-attention layer parameters, unfreeze all decoder parameters and apply adversarial loss to adjust all decoder parameters. Dynamically adjust the pruning threshold based on gradient direction similarity. Add a scope encoding vector to the decoder embedding layer to achieve explicit encoding of scope hierarchy. Apply a coverage penalty term to the decoder attention to adjust the overall model. Obtain the adjusted overall model.

2. The method for training large models for low-code development according to claim 1, characterized in that, The output code is parsed into an abstract syntax tree, which in turn generates a hierarchical code sequence, specifically including: The output code is parsed into an abstract syntax tree, and a hierarchical code sequence with hierarchical tags is generated by depth-first traversal. The hierarchy of each code block in the hierarchical code sequence is marked, so that the large model can perceive the code structure through the delimiter tags.

3. The method for training large models for low-code development according to claim 1, characterized in that, Data augmentation operations are performed on the sample codes corresponding to the sample data in the training set, specifically including: Parse the abstract syntax tree corresponding to the sample code and identify preset identifiers; the preset identifiers include at least: user-defined variables, function names, and class names; generate random alternative names that conform to preset rules for each preset identifier; Using hash mapping or random strings ensures that references within the same scope are synchronously replaced and updated. Perform syntax structure pattern matching on the abstract syntax tree, and replace the subtrees that successfully match the abstract syntax tree with the corresponding preset new code.

4. The method for training large models for low-code development according to claim 1, characterized in that, Unfreeze the encoder's self-attention layer parameters, freeze the decoder parameters, and set the alignment loss function between the input data and the code sequence as the objective function. Specifically, this includes: Obtain the hierarchical consistency loss and syntax keyword matching loss; Based on the hierarchical consistency loss and syntax keyword matching loss, the code error constraint loss is calculated. By utilizing code error constraint loss and the hidden state representation of the input data after encoder processing, the alignment loss function between the input data and the code sequence is calculated.

5. The method for training large models for low-code development according to claim 1, characterized in that, Adversarial loss is used to adjust all parameters of the decoder, specifically including: Obtain cross-entropy loss and adversarial loss; The total loss function is obtained by weighted summation of cross-entropy loss and adversarial loss; in the adversarial loss network, a syntax validator is used as the discriminator.

6. The method for training large models for low-code development according to claim 1, characterized in that, The cropping threshold is dynamically adjusted based on gradient direction similarity, specifically including: Through the formula: Calculate the clipping threshold ;in, For the first The gradient vector at each time step. For the first The gradient vector at each time step. for transpose, It is the L2 norm. This is the scaling factor.

7. A large model training device for low-code development, characterized in that, The device includes: processor; And a memory having executable code stored thereon, which, when executed, causes the processor to perform a large model training method for low-code development as described in any one of claims 1-6.

8. A non-volatile computer storage medium, characterized in that, It stores computer instructions, which, when executed, implement a large model training method for low-code development as described in any one of claims 1-6.

Images