Methods and systems for code pre-training and generation based on structural variant screening

Abstract

The application belongs to the field of information technology and relates to a code pre-training and generation method and system based on structural variant screening. The method comprises the following steps: analyzing source code, extracting an abstract syntax tree thereof, and expanding the abstract syntax tree into a sequential readable structural path encoding sequence; based on the structural path encoding sequence, generating multiple structural variants under the premise of keeping original semantics unchanged; pre-training a code generation model by using the multiple structural variants, and dynamically selecting a suitable structural variant according to a model state in a training process; and generating code by using the pre-trained code generation model according to user requirements. The application can effectively utilize the structural space of a syntax tree in a training stage, realize dynamic sampling and adaptive optimization of structural diversity input, can be widely applied to generation tasks such as function generation, code completion and code repair, and has strong practical engineering popularization value.

Description

Technical Field

[0001] This invention belongs to the field of information technology, specifically relating to a method and system for code pre-training and generation based on structural variant screening. Background Technology

[0002] With the continuous development of artificial intelligence technology, language model-based code generation methods have gradually become an important technical direction for intelligent software development. In recent years, numerous pre-trained models for programming languages, such as CodeBERT, CodeT5, Codex, CodeGen, and Qwen2.5, have been widely used in tasks such as code completion, function generation, and automatic repair, and have achieved significant progress in multiple benchmark tests. Most of these models draw on the ideas of natural language modeling, treating source code as a sequence of pure basic semantic units (tokens) for modeling. By pre-training model parameters on large-scale open-source code corpora, they achieve implicit learning of program semantics and structure.

[0003] However, unlike natural languages, programming languages ​​possess explicit grammatical rules, nested structures, and hierarchical semantics. During compilation or static analysis, source code is often parsed into an Abstract Syntax Tree (AST), which clearly reveals the hierarchical relationships between statements, expressions, and control structures within the program. Treating code as a typical natural language problem, encoding the program merely as a flat sequence of tokens, may cause the model to overlook its original structural boundaries and control dependencies, thus affecting the syntactic integrity and structural consistency of the generated code. This "lack of structural information" has become a key constraint in current code modeling based on the Transformer architecture.

[0004] To address the issue of structural information loss, existing research has attempted to introduce explicit structural annotations into the input. For example, node types, indentation levels, or code block boundaries in the syntax tree are encoded as additional tokens, or graph neural networks are used to model the AST structure to enhance structural representation capabilities. These methods have improved the model's ability to recognize local syntactic structures to some extent, resulting in improved local semantics and syntactic matching in the generated results. However, most of these methods rely on static structural information to construct a single form of input representation, that is, selecting a fixed traversal order (such as preorder, inorder, breadth-first search, etc.) from the syntax tree to linearize the tree structure as part of the training data. On the one hand, the traversal method of the syntax tree is not unique; the same source code can be structurally represented by multiple semantically equivalent path representations. Using only a single path makes it difficult to fully capture the diversity inherent in the syntactic structure, limiting the model's generalization ability to different structural transformations and hindering its robustness to structural variations.

[0005] On the other hand, some studies attempt to generate transformed and diversified samples through data augmentation techniques to expand the coverage and structural complexity of training data. Currently, mainstream data augmentation methods often focus on perturbations at the code surface level, such as identifier replacement, comment modification, formatting adjustments, or statement-level cutting and splicing. These transformations are mainly based on token-level or local semantic substitution, making it difficult to access the complete syntactic structure behind the code. Augmentation methods truly based on syntax tree structures and capable of systematically generating semantically equivalent representations remain extremely limited. This results in existing methods, while expanding sample diversity, failing to enhance the model's adaptability to structural differences and failing to tap the potential dynamic role of syntactic structure during training.

[0006] Furthermore, current training paradigms generally lack structure selection mechanisms based on model state. Once diverse structural samples are introduced, existing methods often uniformly incorporate them into the training process, lacking evaluation and selection of the training value of structural variants. While different structural variants may be semantically equivalent, their training value for the model can vary significantly. Some structural paths may better suit the current model's learning preferences and be more easily learned efficiently; while other structures may be too complex or incompatible with the current model's representation space, increasing the training burden or even causing convergence instability. In addition, indiscriminately introducing all structural transformation samples can lead to redundant, repetitive, and even mutually interfering training data, thereby weakening the model's ability to capture key structural patterns. This lack of structure-aware sample selection not only reduces training efficiency but also obscures the learning signals of key structural patterns.

[0007] Fundamentally, the "one-to-many" representation capability of program syntax structures—that is, a function or code snippet can be transformed into multiple structurally equivalent but different syntax tree traversal sequences (i.e., structural variants) without changing the semantics—provides a vast space for diversity in training data construction. Therefore, how to leverage the combinatorial potential of syntax structures to rationally generate these equivalent structural variants, and how to adaptively control the structural input during training to construct a training method that co-optimizes the model and structural data, has become a key issue driving further breakthroughs in code generation pre-training capabilities. Unlike traditional static input or random augmentation methods, this integrated mechanism of "structure generation—structure selection—structure learning" will significantly improve training efficiency and the model's structural understanding ability.

[0008] In summary, existing language model-based code generation methods have the following three key limitations in training data structure modeling: (1) Existing models are not good at capturing structural information. Some methods treat code as a normal natural language problem, but ignore the syntactic structure information that constitutes the code.

[0009] (2) Existing methods usually only consider one traversal method and lack comprehensive modeling of equivalent traversal paths in the code syntax structure, resulting in a limited learning space for structural semantics and difficulty in generalizing the model to unseen structural changes. (3) During the training process, it is impossible to adaptively select the structural input that is most valuable for the current model to learn, which can easily introduce redundant or invalid samples to interfere with the training process and affect the stability and learning efficiency of the model.

[0010] Therefore, how to construct structurally diverse samples by combining the generation rules of grammatical structures and encode them into input sequences that can be perceived by the model, while constructing a data and model collaborative selection mechanism during the training process to achieve dynamic adaptation of training sample structures, thereby maximizing the potential gains brought by structural diversity, has become a key problem that urgently needs to be solved in the development of current structure-aware code generation models. Summary of the Invention

[0011] To address the limitations of existing large language models in code generation tasks, this invention provides a method and system for code pre-training and generation based on structural variant filtering. This method effectively utilizes the structural space of the syntax tree during the training phase, enabling dynamic sampling and adaptive optimization of structurally diverse inputs.

[0012] This invention proposes a syntax tree-driven structural variant generation mechanism. Based on the Abstract Syntax Tree (AST) structure of the source code, it systematically generates multiple semantically equivalent but structurally different input sequences. Unlike traditional approaches that use only a single static traversal method, this invention leverages the multi-path characteristics of syntax trees, constructing a set of traversal paths covering different structural branches and grammatical patterns, focusing on dimensions such as node expansion order and subtree selection strategies. These paths can all be reconstructed into valid grammatical structures, possess semantic consistency, and constitute a diverse space of structured training samples. Furthermore, to enhance structure awareness, this invention encodes the generated paths into input sequences with structural cues, preserving information such as node type, level, and grammatical role. This allows the model to receive more structurally expressive signals during the pre-training phase, significantly improving its ability to model grammatical organization, control boundaries, and semantic levels. This mechanism can expand to a large number of semantically preserved training samples even with limited original data, achieving sufficient coverage and enhancement of the structural distribution of the training data.

[0013] This invention further proposes a structure selection and adaptation mechanism during the training phase, used to dynamically identify and select the most valuable input path for the current model training from multiple candidate structure samples. This mechanism evaluates and filters candidate structures at each training phase, selecting the most helpful structural variants for model learning from the candidate space and incorporating them into the training process, thus achieving hierarchical control of the training data's structure. Through dynamic structure selection, this invention not only reduces the interference of invalid structural samples on training results but also enhances the model's ability to identify and adapt to different structural forms, enabling model training to achieve faster convergence speed and stronger structure generalization ability while maintaining data diversity.

[0014] The technical solution adopted in this invention is as follows: A method for code pre-training and generation based on structural variant filtering includes the following steps: Parse the source code, extract its abstract syntax tree, and expand the abstract syntax tree into a sequentially readable structure path encoding sequence; Based on the structure path encoding sequence, multiple structure variants are generated while maintaining the original semantics; The code generation model is pre-trained using the various structural variants, and the appropriate structural variant is dynamically selected based on the model state during the training process; Based on user needs, code is generated using the pre-trained code generation model.

[0015] Furthermore, the abstract syntax tree describes the composition logic of the source code in a tree structure. Each node in the tree corresponds to a basic syntax unit, and the edges in the tree represent the nesting or subordinate relationship between syntax units.

[0016] Furthermore, the structure path encoding sequence uses syntactic units as the basic building blocks and adds structural feature information, which includes the type, position, and nesting level of each node; a structure tagging mechanism is introduced during the encoding process: for occluded, deleted, or hidden structural paragraphs, tagging information is embedded in a placeholder manner.

[0017] Furthermore, the generation of various structural variants while preserving the original semantics includes: The structural path is deformed, clipped, and occluded from multiple dimensions to generate various structural variants. All structural transformation processes are carried out under structural constraints to ensure the integrity and logical consistency of the sample semantics. After generating each structural path, the system automatically performs structural restoration and semantic verification to ensure that it can be restored into a valid abstract syntax tree and maintains semantic equivalence with the original code.

[0018] Furthermore, the step of dynamically selecting appropriate structural variants based on the model state during training includes: Starting from the overall training effect, the value of candidate structures in terms of representation diversity, training contribution and context coverage is comprehensively considered. At the same time, the performance feedback of different paths in the model in historical training is combined to form a priority evaluation of structural paths. Based on the priority evaluation, during each round of training, the most suitable one or more structural paths are selected from the structural candidate set to participate in the training, thereby achieving precise scheduling of structural inputs.

[0019] Furthermore, the pre-training employs an adaptive training strategy, enabling the training process to automatically adjust the learning focus based on the input structural features, allowing the model to learn different types of structural representations and generation capabilities at different training stages; the adaptive training strategy includes: In the early stages of training, sample paths with complete structures and rich contexts are introduced first to guide the model to master basic structural awareness and sequential modeling capabilities. In the later stages of training, structural paths including structural missing, occlusion, and predicted occupancy are gradually introduced to enhance the model's ability in structural reasoning, completion, and generation, thus constructing a learning process that evolves progressively from structural awareness to structural generation.

[0020] Furthermore, the pre-training supports flexible configuration of structural task objectives. Based on the labeling information of different structural segments in the samples, the model automatically adjusts the task focus, ensuring that the training objective always maintains consistency and guidance with the structural input based on the linkage perception between structural labels and input states.

[0021] A system for code pre-training and generation based on structural variant filtering, comprising: The structure-aware representation module is used to parse source code, extract its abstract syntax tree, and expand the abstract syntax tree into a sequentially readable sequence of structure path codes. The structure-enhanced data generation module is used to generate multiple structural variants based on the structure path encoding sequence while keeping the original semantics unchanged. The structure selection and optimization training module is used to pre-train the code generation model using the various structure variants, and dynamically select the appropriate structure variant according to the model state during the training process. The code generation inference module is used to generate code based on user requirements using the pre-trained code generation model.

[0022] This invention has the following beneficial effects in the synergistic optimization of structural modeling capability and training effect: (1) Significantly enhanced structural modeling capability: Compared with the traditional linear token input method, this invention can fully explore the traversal space of the grammatical structure and train the model using multiple semantically equivalent structural forms, thereby improving the robustness and expressive ability of the model to structural variations. (2) Improved training efficiency and quality: Through the dynamic structure selection mechanism, the model is guided to select the structure sample with the most learning value, which effectively reduces training redundancy, improves gradient convergence speed, and alleviates the training instability caused by structure perturbation. (3) Wide adaptability and practicality: The method of this invention is applicable to pre-trained language models of various architectures and can be used as a general component of the structure-aware training paradigm. It is widely used in generation tasks such as function generation, code completion, and code repair, and has strong practical engineering promotion value. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating the steps of code pre-training and generation based on structural variant filtering.

[0024] Figure 2 This is a schematic diagram illustrating the processing steps of a code pre-training and generation method based on structural variant filtering.

[0025] Figure 3 This is a module composition diagram of the system based on code pre-training and generation using structural variant filtering. Detailed Implementation

[0026] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below through specific implementation examples.

[0027] This invention proposes a method and system for code pre-training and generation based on structural variant selection. By constructing semantically equivalent syntax tree structural variants and introducing a structural selection mechanism during training, it achieves a significant improvement in the efficiency of deep modeling and learning of code syntax structure. This invention designs a system to automatically solve key problems such as sparse code training data structure, weak structural alignment, and lack of structure-aware control during the training process.

[0028] The method for code pre-training and generation based on structural variant filtering of the present invention, such as Figure 1 , Figure 2 As shown, it includes the following steps: Parse the source code, extract its abstract syntax tree, and expand the abstract syntax tree into a sequentially readable structure path encoding sequence; Based on the structure path encoding sequence, multiple structure variants are generated while maintaining the original semantics; The code generation model is pre-trained using the various structural variants, and the appropriate structural variant is dynamically selected based on the model state during the training process; Based on user needs, code is generated using the pre-trained code generation model.

[0029] In one embodiment, the abstract syntax tree describes the composition logic of the source code in a tree structure, where each node in the tree corresponds to a basic syntax unit, and the edges in the tree represent the nesting or subordinate relationships between syntax units.

[0030] In one embodiment, the structural path encoding sequence uses syntactic units as the basic building blocks and adds structural feature information, wherein the structural feature information includes the type, position, and nesting level of each node; a structural tagging mechanism is introduced during the encoding process: for occluded, deleted, or hidden structural paragraphs, tagging information is embedded in a placeholder manner.

[0031] In one embodiment, generating multiple structural variants while preserving the original semantics includes: The structural path is deformed, clipped, and occluded from multiple dimensions to generate various structural variants. All structural transformation processes are carried out under structural constraints to ensure the integrity and logical consistency of the sample semantics. After generating each structural path, the system automatically performs structural restoration and semantic verification to ensure that it can be restored into a valid abstract syntax tree and maintains semantic equivalence with the original code.

[0032] In one embodiment, dynamically selecting an appropriate structural variant based on the model state during training includes: Starting from the overall training effect, the value of candidate structures in terms of representation diversity, training contribution and context coverage is comprehensively considered. At the same time, the performance feedback of different paths in the model in historical training is combined to form a priority evaluation of structural paths. Based on the priority evaluation, during each round of training, the most suitable one or more structural paths are selected from the structural candidate set to participate in the training, thereby achieving precise scheduling of structural inputs.

[0033] In one embodiment, the pre-training employs an adaptive training strategy, enabling the training process to automatically adjust the learning focus based on the input structural features, allowing the model to learn different types of structural representations and generation capabilities at different training stages; the adaptive training strategy includes: In the early stages of training, sample paths with complete structures and rich contexts are introduced first to guide the model to master basic structural awareness and sequential modeling capabilities. In the later stages of training, structural paths including structural missing, occlusion, and predicted occupancy are gradually introduced to enhance the model's ability in structural reasoning, completion, and generation, thus constructing a learning process that evolves progressively from structural awareness to structural generation.

[0034] In one embodiment, the pre-training supports flexible configuration of structural task objectives. Based on the labeling information of different structural segments in the samples, the model automatically adjusts the task focus, ensuring that the training objective always maintains consistency and guidance with the structural input based on the linkage perception between structural labels and input states.

[0035] The system for code pre-training and generation based on structural variant filtering of the present invention, such as Figure 3 As shown, the system comprises a structure-aware representation module, a structure-enhanced data generation module, a structure selection and optimization training module, and a code generation inference module, possessing structure-level pre-training and generalization capabilities. In this system, the structure-aware representation module parses source code, extracts its abstract syntax tree, and expands the abstract syntax tree into a sequentially readable structure path encoding sequence; the structure-enhanced data generation module generates multiple structure variants based on the structure path encoding sequence while preserving the original semantics; the structure selection and optimization training module pre-trains the code generation model using these multiple structure variants, dynamically selecting the appropriate structure variant based on the model's state during training; and the code generation inference module generates code based on user requirements using the pre-trained code generation model. The system's input and output are described below: At the input end, the user provides the original code. The system first parses the source code and extracts its Abstract Syntax Tree (AST). It then performs structural analysis and traversal path construction on the AST, deriving multiple semantically equivalent but structurally diverse representation sequences from each source function. These sequences are further encoded into a structure-aware input format for constructing structure-enhanced pre-training samples.

[0036] On the output side, the pre-trained and optimized code generation model can accept code snippets, code hints, natural language descriptions, etc. as input, and generate the required code content based on the representation capabilities obtained from structure-aware pre-training. The model's output not only maintains correctness in local syntax, but also has consistency and interpretability at the overall structural dependency level.

[0037] In order to improve the model's ability to model structural representations and the convergence quality of the training process, this invention specifically includes the following four implementation points: (1) a structure-aware encoding method; (2) a data generation method for structural variants; (3) a structural variant selection and adaptive optimization mechanism; and (4) a reasoning and generation method for structural enhancement models.

[0038] I. Structure-aware coding method.

[0039] In code pre-training tasks, the structural representation of training samples has a decisive impact on the model's final learned structural understanding and generation performance. Traditional code generation models often use flattened token sequences as training input, linearly encoding complete code snippets and feeding them into the model for learning, without retaining any explicit structural cues or semantic hierarchical information. While this approach is simple to implement, it ignores the inherent syntactic structure characteristics of programs, making it difficult to reveal important features such as the syntactic organization, scope, and nesting relationships behind the code, thus limiting the model's ability to model structural variations and hierarchical semantics.

[0040] To overcome the aforementioned problems, this invention proposes a unified structure-aware encoding method. Unlike traditional methods that only use flat token sequences as model input, this invention starts from the abstract syntactic structure of the code and designs a structure-aware encoding mechanism. This aims to maintain the continuity of the input sequence while explicitly embedding the syntactic structure information of the code. This allows the model to maximize the preservation of the program's syntactic hierarchy information during the training phase and explicitly identify structural breakpoints or positions to be generated, thereby improving the expression and generation capabilities of pre-trained code models under complex syntactic structures. The method first receives source code text provided by the user, with input being arbitrary code text (supporting languages ​​with clear syntactic definitions, such as Python). A syntactic structure parsing module (such as a general syntax analysis tool) transforms this text into an abstract syntactic representation with hierarchical features. This representation describes the compositional logic of the source code in a tree structure, where each node corresponds to a basic syntactic unit (such as variable declarations, expressions, control statements, etc.), and the edges in the tree depict the nesting or dependency relationships between these syntactic units. This structure not only completely preserves the syntactic construction of the source program but also clearly shows the hierarchy and scope boundaries between code blocks.

[0041] Building upon this foundation, this invention presents a method for constructing structured linear sequences, which expands a syntax tree into a sequentially readable sequence of structured path codes as model input. Unlike traditional tokenized plain text encoding, this structured sequence is built upon syntactic units, supplemented with structural features such as the type, position, and nesting level of each node. This allows the model to perceive the organization and logical division of the code when reading the input. To enhance the model's adaptability in situations involving structural gaps or code completion, this method introduces a structural labeling mechanism during encoding: for occluded, deleted, or hidden structural segments, placeholder information is embedded to indicate whether this is a "structural break" or a "segment to be predicted." Through this design, the model can differentiate between semantically continuous and structurally missing locations, enabling more targeted structural completion and reconstruction reasoning.

[0042] In summary, the structure path encoding sequence has the following significant advantages: (1) Structural integrity: The structure path encoding sequence can fully express the syntactic organization of the source code while maintaining readability, so that the model can not only obtain the surface lexical, but also understand the construction logic of the code structure; (2) Cross-language adaptability: As long as it is adapted according to the syntax tree construction rules, this serialization method can be extended to a variety of mainstream programming languages, which is convenient for the system to uniformly model in multi-language scenarios; (3) Structural reversibility: This encoding is not only used for model input, but can also be backed to the original syntax tree through the structure restoration mechanism, which has good interpretability and debuggability; (4) Training flexibility: Any node in this structure sequence can be used as a potential "occlusion target" or "prediction position", providing a unified operation unit and dynamic control basis for subsequent occlusion modeling, structure insertion and other tasks.

[0043] Overall, this input encoding method introduces a structural dimension information channel on top of the original token, providing the model with a dual modeling foundation of explicit structural cues and implicit structural context. Furthermore, the structure-aware encoding method designed in this invention is highly synergistic with subsequent mechanisms such as structural variant data generation and sample selection. This method not only enables the model to recover structurally missing content but also provides a unified input basis for subsequent training stages, significantly enhancing the model's ability to learn and generate code structures.

[0044] II. Data generation methods for structural variants.

[0045] This invention addresses the problems of scarce structural samples and limited grammatical patterns in code pre-training tasks by proposing a data generation method oriented towards structural variants. Starting with source code and combining it with the structural information revealed by the Abstract Syntax Tree (AST), this method constructs multiple grammatically distinct input paths through structural-level transformations while preserving the original semantics, thus forming a representative and comprehensive set of candidate structural samples.

[0046] Unlike traditional data augmentation methods that primarily rely on surface-level word order, this invention emphasizes a structure-centric variant generation strategy. During variant sample generation, the source code is first converted into an abstract syntax tree and further encoded into a linearized structural path representation, serving as the foundation for subsequent sample construction. Based on this, the structural path is flexibly adjusted across multiple dimensions, including syntactic hierarchy, node order, traversal strategy, and structural visibility. Common operations include: transformation operations to change the organization of the syntactic structure or node relationships to explore different structural arrangements; pruning operations to moderately simplify local structures while preserving the main semantic skeleton, generating input samples of different granularities; and occlusion operations to simulate structural gaps or unpredictable scenarios by setting placeholder regions or hidden segments in the structural path. This systematically generates multiple structural variants. All structural transformation processes are performed under structural constraints, ensuring the semantic integrity and logical consistency of the samples. After generating each structural path, the system automatically performs structural reconstruction and semantic verification to ensure it can be reconstructed into a valid syntax tree and maintains semantic equivalence with the original code. The core concept of this mechanism is to fully utilize the expressive flexibility of program structure, and without affecting the original semantics, to explore structural diversity pairs and construct semantically equivalent but structurally rich input data, thereby significantly improving the model's ability to perceive and generalize code structure.

[0047] Furthermore, the generated samples explicitly label the locations of structural gaps or growths, enabling them to guide the model in structure prediction, placeholder filling, and pattern completion during training. This structure-embedded data construction method not only enhances the model's robustness to structural gaps but also provides controllable sample distribution support for the structure selection mechanism during training. All samples are stored in a uniform format and can be scheduled on demand during the training phase, achieving training with structural diversity and adaptive learning based on structural preferences.

[0048] In summary, this invention generates data through syntactic structure, systematically expanding the structural space of training samples while preserving the original semantics, and explicitly encoding structural information into the input data, laying the foundation for improving the model's structural understanding capabilities. Simultaneously, this method provides a candidate sample pool, a crucial pre-step for achieving adaptive structural optimization training. This method effectively overcomes the shortcomings of existing pre-training methods, such as "single code structure and poor syntactic styles," providing a low-cost, highly structurally diverse sample construction mechanism.

[0049] III. Structural variant screening and adaptive optimization mechanism.

[0050] Introducing structure-aware encoding and diverse structural variants during pre-training significantly expands the model's structural cognitive boundaries. However, structural diversity itself does not directly lead to performance improvements. The real challenge lies in dynamically selecting the most valuable structural input from multiple semantically consistent but structurally diverse candidates, and enabling the model training process to adapt to task variations under different structural forms. To address this, this invention proposes a training data selection and optimization mechanism oriented towards structural variants, balancing structural expressiveness and training efficiency, and constructing a structure-driven dynamic pre-training process.

[0051] 3.1 Structural path filtering mechanism.

[0052] To address the issue of multiple semantically equivalent structural variants for each sample, this invention further proposes a structural path selection mechanism to dynamically select appropriate structural paths based on the model's state during training. This mechanism does not rely on a specific variant generation method but rather considers the overall training effect, comprehensively evaluating the value of candidate structures in terms of representational diversity, training contribution, and contextual coverage. It also incorporates dynamic signals such as gradient contribution, prediction bias, and convergence speed from the model's historical training performance on different paths to form a priority evaluation of structural paths.

[0053] For example, after generating multiple semantically equivalent structural paths, the system first extracts the structural statistical features of each path, including node type distribution, hierarchy depth, and path length, to reflect structural differences. Subsequently, during training, the system records the model's learning performance on each path sample, such as loss trends, prediction consistency, and convergence stability. Based on these static and dynamic features, the system comprehensively evaluates each candidate path by setting weights or scoring rules, thereby determining its priority in the current training stage. This evaluation process can be implemented using any form of weighting or ranking mechanism, and the evaluation metrics can be flexibly adjusted according to different model architectures or task objectives. For example, in the early training stage, the system can prioritize paths with high structural coverage and large sample differences to improve the model's structural recognition ability; in the later training stage, it can prioritize paths with stable convergence and high gradient contributions based on model feedback to improve training efficiency and stability.

[0054] Through the above methods, a dynamically adjustable structural path priority evaluation framework can be formed, enabling the system to adaptively select structural inputs based on the model state, balancing structural diversity and training effectiveness. Based on the above evaluation, during each round of training, one or more suitable structural paths can be selected from the candidate set to participate in training, thereby achieving precise scheduling of structural inputs. This strategy avoids excessive repetition and structural redundancy during training, while also ensuring a reasonable distribution of different structural features during training, effectively improving the model's coverage of structural variants and learning efficiency. Furthermore, this mechanism enables the model to "perceive structural input—feedback on training results—optimize structural selection," achieving proactive screening and efficient utilization of structural data. This mechanism can also serve as an infrastructure module for other strategies such as structural enhancement, contrastive training, and data pruning, coordinating with them or replacing manual, experience-based screening schemes to support a more scalable and adaptive training process. Through this mechanism, the model not only obtains the structural information itself but also gradually builds an intrinsic modeling ability for structural preferences.

[0055] 3.2 Adaptive training strategy.

[0056] With the introduction of structural diversity, the learning tasks faced by the model will vary in difficulty and form. To adapt to this diversity, this invention designs a training task adaptive adjustment mechanism, enabling the training process to automatically adjust the learning focus according to the input structural features.

[0057] The core idea of ​​this mechanism is that the model should learn different types of structural representation and generation capabilities at different training stages. For example, in the early stages of training, the system prioritizes introducing sample paths with complete structures and rich context to guide the model to master basic structural awareness and sequential modeling capabilities; while in the later stages of training, structural paths containing structural missing parts, occlusions, and prediction placeholders are gradually introduced to strengthen the model's capabilities in structural reasoning, completion, and generation, thus constructing a learning process that evolves progressively from "structural awareness" to "structural generation".

[0058] Furthermore, the system supports flexible configuration of structural task objectives. When constructing training samples, the system assigns corresponding task labels to various structural fragments (such as function bodies, control statements, or named regions), such as structural prediction, named recovery, or semantic concatenation. During training, the system dynamically adjusts the target weights and activation strategies of different training sub-tasks based on the type of structural input, labeling information, and model state. When the model detects an increase in the proportion of a certain type of structural feature in the input sample or a missing feature, it can automatically adjust the learning intensity of the corresponding task to ensure that the training objective matches the structural input. This adaptability of the training objective does not depend on specific label types but is based on the interconnected perception of structural labels and input states, ensuring that the training objective always maintains consistency and guidance with the structural input.

[0059] For example, when a large number of occluded code segments are detected in a sample, the system automatically strengthens the structural completion task; when the sample is predominantly composed of complete structures, the optimization weights for structural consistency and generation accuracy are increased. The configuration process can be implemented through mechanisms such as task scheduling, loss weighting, or adaptive sampling, without being limited to a specific algorithm. Through this flexible configuration mechanism, the system can maintain a dynamic balance between task objectives under different structural features and training stages, thereby improving the model's adaptability to structural diversity and generation stability.

[0060] 3.3 Advantages of the overall framework.

[0061] By introducing a structural path selection mechanism and an adaptive training strategy, this invention achieves organic linkage between structural variant generation, variant sample selection, and structure-driven training. Compared to traditional training processes, this framework not only improves the utilization rate of structural diversity in training samples but also significantly enhances the model's adaptability to different structural morphologies. More importantly, this framework establishes a feedback loop based on structural signals, enabling the model to actively participate in the data selection process, adjust training data and optimization strategies according to its own state, and drive structure-guided training to evolve into structure-adaptive optimization, comprehensively improving the model's understanding and generation quality under complex structural conditions.

[0062] This framework does not rely on fixed data structures or a single selection strategy. Instead, it provides a general collaborative framework for model participation in structure evaluation, path selection, and goal control, and is compatible with various training tasks and data construction strategies. Regardless of how the structure path is generated, the model can dynamically select and adjust based on its own feedback, thereby supporting a self-optimizing training paradigm that is "guided by model feedback and based on structure paths."

[0063] The design philosophy of this framework is to avoid pre-setting structure construction methods and relying on specific labeling systems. Instead, it uses a unified path representation and model feedback signals to drive an adaptable and scalable structure selection and training process. It not only supports existing structure sample selection techniques but also provides an integrated interface for future more complex structure control mechanisms (such as reinforcement learning scheduling, search-guided generation, and policy-learning data sampling).

[0064] IV. Evaluation of the technical effects of the present invention.

[0065] Finally, to verify the practical benefits of this invention in code generation tasks, the generation performance of the model under three input structure forms was systematically compared using the MBPP dataset as a benchmark. This experiment designed three comparison settings based on the structural modeling method of the model training data, and uniformly adopted the Qwen2.5-Coder-0.5B model architecture for training and evaluation: Option 1 (Baseline): The source code is directly modeled as a pure token sequence without any structural information, representing the traditional code pre-training method.

[0066] Option 2 (Structure Encoding): Extract code structure boundary information based on abstract syntax tree, encode each token as its position and type in the syntax structure, realize explicit structure-aware input, but do not introduce structure variants or path filtering mechanisms.

[0067] Scheme 3 (Method of this invention): Based on structural encoding, a structural variant generation strategy is introduced to construct multiple grammatical paths under the same semantic goal. Combined with model feedback, structural path selection and adaptive optimization of training tasks are implemented to achieve enhanced diversity and selection control driven by structure.

[0068] The experimental design aims to systematically evaluate the role of structural modeling in code pre-training, and further verify the effects of structural variant generation and selection, and adaptive training process on improving the model's structural understanding and generation quality.

[0069] First, under Scheme 1, the model achieved a pass@1 score of 38.6% and a pass@5 score of 48.8%. This result represents the average performance level of traditional encoding methods. Building on this, Scheme 2, which introduces structure encoding to enhance the model's awareness of structural locations in the input, improved the pass@1 score to 42.1% and the pass@5 score to 49.5%, demonstrating the positive benefit of structure-aware encoding on the model's learning ability, particularly in terms of code generation accuracy and semantic constraint consistency.

[0070] Furthermore, this invention grows multiple structural paths at each abstract syntax tree node to simulate the natural distribution of structural diversity, thereby improving the model's adaptability to complex grammatical configurations. In the initial experiment without introducing dynamic structural path selection, ten paths (S-10) were constructed as training inputs. In the experiment with dynamic structural path selection, three paths (S-3) were constructed as training inputs, one of which was the highest-scoring path, and the remaining two were randomly selected. The results show that with S-3 paths, the model's pass@1 was 50% and pass@5 was 60.0%; while with S-10 random paths, the performance further decreased to 33.1% (pass@1) and 42.9% (pass@5). Despite introducing richer structural diversity, the model performance declined in the absence of a structural selection mechanism, indicating that more structural expansion is not necessarily better, and blindly introducing paths increases the learning difficulty of the model and may even cause structural bias interference. This phenomenon also verifies that the training of structure-aware models requires effective structural preference control strategies.

[0071] Based on the above experiments, the following conclusions can be drawn: Compared with traditional encoded sequence input, the structure-aware encoding method proposed in this invention has shown a significant improvement in model learning performance without significantly increasing training resource costs. While the structural diversity introduced by the data generation method oriented towards structural variants slightly reduces performance under unfiltered conditions, it provides greater operational space and upper limit potential for subsequent structural preference modeling and inference optimization.

[0072] As software development becomes increasingly complex, developers face more and more challenges. Code generation models have effectively improved the efficiency and quality of code generation, especially in tasks such as function generation and code fixing, reducing human intervention and increasing development efficiency. However, as the scale of model parameters continues to expand, the demand for training data and computing resources also increases. The method of this invention can utilize existing data more efficiently, improving model training performance and thus meeting the ever-growing needs of software development.

[0073] It should be understood that the methods and systems disclosed in the above embodiments of the present invention can be implemented in other ways. For example, the above module division can be implemented in other ways, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Each step and module in the present invention can be implemented in the form of software functional units and can be stored in a computer-readable storage medium, including several instructions to cause a computer device to execute some or all of the steps of the method described in the present invention. For example, one embodiment of the present invention provides a computer device (computer, server, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for executing each step of the method of the present invention. For example, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk, etc.) storing a computer program, which, when executed by a computer, implements each step of the method of the present invention. For example, another embodiment of the present invention provides a computer program product including a computer program, which, when executed by a computer, implements the steps of the method of the present invention.

[0074] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and to implement it accordingly. Those skilled in the art will understand that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the content disclosed in the embodiments of this specification; the scope of protection of the present invention is defined by the claims.

Claims

1. A method for code pre-training and generation based on structural variant selection, characterized in that, Includes the following steps: Parse the source code, extract its abstract syntax tree, and expand the abstract syntax tree into a sequentially readable structure path encoding sequence; Based on the structure path encoding sequence, multiple structure variants are generated while maintaining the original semantics; The code generation model is pre-trained using the various structural variants, and the appropriate structural variant is dynamically selected based on the model state during the training process; Based on user needs, code is generated using the pre-trained code generation model.

2. The method according to claim 1, characterized in that, The abstract syntax tree describes the composition logic of the source code in a tree structure. Each node in the tree corresponds to a basic syntax unit, and the edges in the tree represent the nesting or subordinate relationship between syntax units.

3. The method according to claim 1, characterized in that, The structure path encoding sequence uses syntactic units as the basic building blocks and adds structural feature information, which includes the type, position, and nesting level of each node. A structural tagging mechanism is introduced during the encoding process: for occluded, deleted, or hidden structural paragraphs, tagging information is embedded in a placeholder manner.

4. The method according to claim 3, characterized in that, The process of generating multiple structural variants while preserving the original semantics includes: The structural path is deformed, clipped, and occluded from multiple dimensions to generate various structural variants. All structural transformation processes are carried out under structural constraints to ensure the integrity and logical consistency of the sample semantics. After generating each structural path, the system automatically performs structural restoration and semantic verification to ensure that it can be restored into a valid abstract syntax tree and maintains semantic equivalence with the original code.

5. The method according to claim 1, characterized in that, The dynamic selection of appropriate structural variants based on the model state during training includes: Starting from the overall training effect, the value of candidate structures in terms of representation diversity, training contribution and context coverage is comprehensively considered. At the same time, the performance feedback of different paths in the model in historical training is combined to form a priority evaluation of structural paths. Based on the priority evaluation, during each round of training, the most suitable one or more structural paths are selected from the structural candidate set to participate in the training, thereby achieving precise scheduling of structural inputs.

6. The method according to claim 1, characterized in that, The pre-training adopts an adaptive training strategy, which enables the training process to automatically adjust the learning focus according to the input structural features, so that the model can learn different types of structural representations and generation capabilities at different training stages. The adaptive training strategy includes: In the early stages of training, sample paths with complete structures and rich contexts are introduced first to guide the model to master basic structural awareness and sequential modeling capabilities. In the later stages of training, structural paths including structural missing, occlusion, and predicted occupancy are gradually introduced to enhance the model's ability in structural reasoning, completion, and generation, thus constructing a learning process that evolves progressively from structural awareness to structural generation.

7. The method according to claim 6, characterized in that, The pre-training supports flexible configuration of structural task objectives. Based on the labeling information of different structural segments in the samples, the model automatically adjusts the task focus. On the basis of the linkage perception between structural labels and input states, it ensures that the training objectives are always consistent and guided with the structural input.

8. A system for code pre-training and generation based on structural variant filtering, characterized in that, include: The structure-aware representation module is used to parse source code, extract its abstract syntax tree, and expand the abstract syntax tree into a sequentially readable sequence of structure path codes. The structure-enhanced data generation module is used to generate multiple structural variants based on the structure path encoding sequence while keeping the original semantics unchanged. The structure selection and optimization training module is used to pre-train the code generation model using the various structure variants, and dynamically select the appropriate structure variant according to the model state during the training process. The code generation inference module is used to generate code based on user requirements using the pre-trained code generation model.

9. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method according to any one of claims 1 to 7.

Images