A germination pre-training method and system for logical structured data modeling

Abstract

This invention discloses a germination-based pre-training method and system for modeling logically structured data, belonging to the field of information technology. The method includes: performing data logic analysis on raw data to obtain a structured representation of the raw data; performing data germination based on the structured representation to obtain logically structured data x; and obtaining several data sample pairs {x} according to the growth path of the logically structured data x during the data germination process. i‑a ,x i‑b}; Based on the data sample pair {x i‑a ,x i‑b The target model is trained to obtain a germination-based pre-trained model. This invention fully explores the logical structure that each piece of data may contain and applies appropriate germination methods to accurately generate the target data content under different contexts, thereby enabling more effective processing of downstream tasks.

Description

Technical Field

[0001] This invention belongs to the field of information technology, specifically relating to an germination-style pre-training method and system for logically structured data modeling. Background Technology

[0002] Logically structured data refers to data forms with explicit logical rules or semantic structures, such as imperative programming language code, compiler intermediate representations, and finite state machine models. Due to its high precision and ease of parsing, this type of data plays a crucial role in scientific computing, knowledge representation, automated design, and data management. Pre-training is a technique that learns general data representations by performing preliminary modeling on large-scale data. Pre-training modeling not only extracts the basic syntactic and semantic information of text data but also provides shared knowledge for various downstream tasks, thereby improving the model's generalization ability and data utilization efficiency. However, existing pre-training techniques are mainly based on unstructured natural language data, focusing on capturing basic syntax, semantics, and contextual relationships, while neglecting the characteristics of strongly structured logical data. For example, logically structured code data not only has syntactic and semantic rules but also includes strict logical control structures, conditional judgments, and hierarchical and accurate requirements such as program flow. This makes it difficult for traditional pre-training methods to fully capture the inherent, strongly correlated logical structure information when processing this type of data. Therefore, designing a pre-training method and system for logically structured data is a crucial issue.

[0003] Existing pre-training techniques primarily model logically structured data in two ways: 1) perceiving the components of the data; and 2) predicting the structured representation of the data. The first method first selects a specific component from the logically structured data, such as identifiers in code. Then, it sets a certain probability to occlude the selected content, where the occluded content can be several tokens or all content belonging to that component. Finally, a deep learning model is used to predict the occluded parts or categories to perceive the components and specific content of the logically structured data, thereby modeling the overall representation of the data using the pre-trained model. The second method first transforms the logically structured data into a special structured sequence, such as the grammatical sequence of code data. Then, based on this structured sequence, it applies pre-training techniques similar to those in natural language processing, such as setting masks or adding noise to the original input and then recovering the form, and uses a deep learning model for prediction. Finally, the generated complete structured sequence representation is reverse-engineered back into the original token form, such as code text, to obtain the final output.

[0004] The main drawback of existing technologies is the lack of sufficient exploration into methods for generating logically structured data. Currently, both the composition of perceptual data and the structured representation of predictive data are generated sequentially from beginning to end, as is common in traditional autoregressive methods. However, for logically structured data, the preceding content may depend on the subsequent content for its determination. For example, adding functionality at the end of code may require additional processing modules in the preceding code. In real-world development scenarios, programmers rarely write all utility functions at once before adding other processing logic; they typically supplement and improve the code incrementally. When generating data only from beginning to end, existing pre-training methods cannot observe the logical representation of the same data from different orders and perspectives. Therefore, it is difficult to effectively learn the underlying logical patterns in the data, resulting in a need for a large amount of data to support the acquisition of internal logical structure information. Furthermore, in training patterns generated only from beginning to end, errors in the model's initial stages may lead to irreversible errors in subsequent results, resulting in model illusions and unreliable outputs.

[0005] In summary, existing methods for pre-training logically structured data primarily guide the pre-training process based on two specific approaches: perceiving the components of the data or predicting the structured representation of the data. However, both approaches still face the challenge of insufficiently exploring the generation methods of logically structured data. This challenge leads to the current requirement for large amounts of data to learn and statistically analyze common logical patterns and representations in pre-training of logically structured data, and makes it possible for pre-emptive errors to result in irreversible errors later. Therefore, in an era of rapid development of large-scale pre-trained models, designing efficient pre-training methods for logically structured data to improve the performance of the final pre-trained model has become a crucial issue. Summary of the Invention

[0006] To address the aforementioned issues, this invention provides a sprouting pre-training method and system for modeling logically structured data. This method fully explores the logical structure that each piece of data may contain and applies appropriate sprouting methods to accurately generate target data content under different contexts, thereby enabling more effective processing of downstream tasks.

[0007] To achieve the above objectives, the technical solution of the present invention includes the following:

[0008] An germination-based pre-training method for modeling logically structured data, the method comprising:

[0009] Perform data logic analysis on the raw data to obtain a structured representation of the raw data;

[0010] Data germination is performed based on the structured representation to obtain logically structured data x. Then, based on the growth path of this logically structured data x during the data germination process, several data sample pairs {x} are obtained. i-a,x i-b}; where the logically structured data x is data with explicit logical rules or semantic structure, x i-a Let x represent the data before germination in the i-th germination process. i-b This represents the data after data x has germinated during the i-th germination process;

[0011] Based on the data sample pair {x i-a ,x i-b The target model is trained to obtain a germination pre-trained model.

[0012] Furthermore, the data structure of the logically structured data x includes: natural language description text or code.

[0013] Furthermore, when the data structure of the logically structured data x is code, based on the data sample, {x} i-a ,x i-b The pre-training of the target model is completed, resulting in the pre-trained target model, including:

[0014] The target model is pre-trained on DeepSeekCoder-Base; wherein the target model is built on the Decoder framework.

[0015] Data sample x i-a Input into the pre-trained target model and combine with data samples x i-b Obtain the loss of the pre-training task based on the germination strategy. And the loss of the next token prediction task The pre-training task based on the germination strategy refers to taking the logical structure representation of data as the object, starting to grow at any position in the structured data until a data form with a complete logical structure is germinated. The germination strategy includes: randomly selecting a node as the node to be grown, gradually generating from left to right, and selecting the node to be grown according to probability.

[0016] Combined with the aforementioned loss and loss The final loss was incurred;

[0017] Backpropagation is performed based on the final loss to update the parameters of the pre-trained target model.

[0018] Furthermore, the loss Where n represents the number of times the logically structured data x is generated, z represents the content of z tokens that need to be predicted in the i-th generation process, and P (θ) Let y represent the probability distribution fitted by the target model, θ represent the parameters of the target model, and y represent the probability distribution fitted by the target model. tRepresents data sample x i-b The content of the t-th node in x pred Represents data sample x i-a The unknown data that needs to be predicted.

[0019] Furthermore, the loss Where k represents the data sample x i-a The number of tokens in P (θ) Let x represent the probability distribution fitted by the target model. t Represents data sample x i-a The t-th token in the array, x 1:t-1 Represents data sample x i-a The first to the (t-1)th tokens in the array.

[0020] Furthermore, the data sample pair {x} i-a ,x i-b After completing the training of the target model and obtaining the germination pre-trained model, the following steps are also included:

[0021] A sprouting strategy is set, which includes: randomly selecting a node as a node to be grown, gradually generating nodes from left to right, and selecting nodes to be grown based on probability.

[0022] Based on the aforementioned germination strategy, select the data to be grown, x. i-a The nodes to be grown;

[0023] Based on the selected nodes to be grown, the germination-based pre-trained model is invoked to obtain the data to be grown, x. i-b And based on the growth data x i-b Re-execute the selection of data to be grown based on the germination strategy. i-a The nodes to be grown will be selected until the set number of germinations is reached.

[0024] An germination-based pre-training system for logically structured data modeling, the system comprising:

[0025] The data transformation module is used to perform data logic analysis on the raw data and obtain a structured representation of the raw data;

[0026] The data germination module is used to perform data germination based on the structured representation to obtain logically structured data x, and to obtain several data sample pairs {x} according to the growth path of the logically structured data x in the data germination process. i-a ,x i-b}; where the logically structured data x is data with explicit logical rules or semantic structure, x i-a Let x represent the data before germination in the i-th germination process.i-b This represents the data after data x has germinated during the i-th germination process;

[0027] The model training module is used to train {x} based on the data samples. i-a ,x i-b The target model is trained to obtain a germination pre-trained model.

[0028] An electronic device includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the germination pre-training method for logically structured data modeling as described above.

[0029] A computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the germination pre-training method for logically structured data modeling as described above.

[0030] A computer program product, characterized in that, when the computer program product is run on a computer device, the computer device executes the germination pre-training method for logically structured data modeling as described above.

[0031] Compared with the prior art, the present invention has at least the following beneficial effects.

[0032] In its design phase, this invention comprehensively reconsidered the issues related to pre-training modeling of logically structured data, resulting in a novel perspective on the germination-based pre-training of logically structured data, allowing it to grow gradually from scratch. Utilizing this new perspective, the invention enhances the exploration of generation methods for individual logically structured data and breaks away from the traditional generation model. It explores as much as possible the growth methods of each germination point for each piece of logically structured data, thereby fully mining the potential logical structure contained in each piece of data. Furthermore, it applies appropriate germination methods to accurately generate the target data content given different contexts, thus enabling more effective processing of downstream tasks. Attached Figure Description

[0033] Figure 1 It is the basic process of an germination-style pre-training method and system for modeling logically structured data. Detailed Implementation

[0034] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below through specific embodiments and accompanying drawings.

[0035] This invention implements a germination-based pre-training method and system for logically structured data. At the input end, the user designs a data germination method based on the serialized logically structured data; at the output end, the model designs a pre-training task on the germination and growth data to guide the model pre-training process. The pre-trained model receives the input logically structured data or related requirement descriptions, and then automatically synthesizes the required target data to meet the user's needs or address the user's modification intentions.

[0036] Specifically, based on fully utilizing the inherent logical relationships in logically structured data and the importance of pre-trained models, this invention designs a system to automatically solve problems related to pre-training and modeling of logically structured data. The input and output of this system are described below:

[0037] At the input end, during the germination stage of logically structured data, the user inputs raw data, and the system serializes and represents it according to the internal logical characteristics of the data, then performs subsequent germination-style growth of structured data. At the output end, after the model pre-training is completed and the application stage begins, the user provides a description of the required logically structured data or data that needs to be edited. When a description of the logically structured data requirement is input, the model automatically generates data that conforms to the requirement description, such as code. When logically structured data that needs to be edited is input, the model automatically edits existing data to meet the new requirements or improve data quality.

[0038] At the output end, the system of this invention trains a targeted model based on the large-scale logically structured data generated from the germination process, and automatically generates target data that meets the requirements based on the knowledge learned by the model.

[0039] To improve upon existing pre-training techniques, this invention: 1. First, designs an germination-based strategy for logically structured data to process the growth of pre-training data; 2. Then, utilizes germination-based pre-training techniques to train deep learning models related to logically structured data; 3. Finally, evaluates the generation results of the germination-based pre-training system for logically structured data. A detailed description follows:

[0040] I. Spawning Strategy for Logically Structured Data

[0041] To better model logically structured data, this invention adopts... Figure 1 The basic process shown applies a germination growth strategy to logically structured data.

[0042] The core idea of ​​the germination strategy for logically structured data is to allow logically structured data to germinate at any location without affecting the integrity and correctness of the overall logic. It leverages the different contexts encountered by individual data samples during germination at different locations to fully learn and perceive the internal logic and structural information of the data. In this invention, logically structured data is defined as a data form with explicit logical rules or semantic structures, such as imperative programming language code, compiler intermediate representations, and finite state machine models. Data germination is defined as the process of data growing from scratch and gradually developing into final data with a complete logical structure, where each germination can begin from any position that still requires growth. For example, in code data with strong logical structure, the code content can grow from the root node of an abstract syntax tree until it becomes complete code content. Germination can be initiated from any syntax tree node, and the resulting code, under the constraints of the code's logical structure, can reconstruct the complete code content.

[0043] according to Figure 1 As shown, data germination involves the analysis of the data's logical structure. The germination strategy takes the representation of the logical structure as its object, and the process can start from any position in the logical structure and grow into complete data. The germination of the logical structure can be divided into the processing of three types of nodes: generated nodes, unknown nodes, and nodes to be predicted. Generated nodes represent nodes in the logical structure whose content is already determined; unknown nodes represent nodes whose logical structure is not yet clear; and nodes to be predicted are a special type of unknown node whose content will be determined during this germination process. Taking code as an example... Figure 1 The first germination in the data growth process represents the expansion of four unknown abstract syntax tree (AST) nodes from the root node, indicating that at least four subtrees of the AST are yet to be generated (germination). In the second germination, a third unknown node is selected as the node to be predicted and germinated. After growth, the logical structure of the data expands to include one generated node and one unknown node awaiting the next growth. In the third germination, the second node of the second level in the entire AST is selected as the node to be predicted and germinated. After growth, the overall logical structure expands again to include three confirmed generated nodes and one unknown node awaiting the next growth. This process continues, and through continuous germination, a complete logical structure representation, i.e., a completed AST structure, is eventually grown, thus generating complete code. A similar germination method can be applied to other logically structured data, and each step of the growth process can determine the content of one or more nodes in the logical structure.

[0044] For each input structured logical data x, a series of specific growth data x1, x2, x3, ..., x can be constructed regarding its germination process. nWhere x1 represents the first germination process of this data, x i This represents the i-th germination process of the data, where n represents the total number of germinations required to obtain data x with a complete logical structure. For each growing data x... i In other words, it can be split into data x before germination. i-a And the data x that completed this germination i-b The germination strategy is also in x i-a Select a node as the node to be predicted and perform prediction. After the prediction is completed, generate x. i-b Ultimately, a raw logically structured data set will be decomposed into many pairs of data samples related to its growth process. For example, the i-th data sample pair can be represented as {x}. i-a ,x i-b These data sample pairs from the growth process can serve as inputs and outputs for the model, enabling the model to learn the logical structure information within the data and accurately generate complete content containing logical constraints during the inference phase.

[0045] This invention decomposes a single data point into a large number of specific germination data points based on the growth path of logically structured data during the germination process. The germination strategy closely aligns with the characteristics of logically structured data because, under strong constraints, the model can adaptively choose an appropriate growth method based on problem requirements without affecting the correctness of the final model generation result. Furthermore, since the growth process can begin from any position, this strategy allows the model to explore the data's growth under various contextual constraints in more ways and along more paths, thereby fully perceiving the constraint information of the data's internal logic and structure.

[0046] II. Sprouting Pre-trained Model

[0047] In this invention, the germination-based pre-training model takes code data with strong logical structure constraints as an example and applies the above-mentioned germination strategy for logically structured data for training and development. The specific development of the germination-based pre-training model can be viewed as a combination of the following three parts: model structure, input / output representation, and pre-training task. It is worth noting that the final germination-based pre-training model is one implementation of the germination strategy for logically structured data proposed in this invention, specifically in one form of logically structured data, i.e., in code.

[0048] 1) Germination-based pre-trained model structure

[0049] The germination strategy for logically structured data involved in this invention is a general approach. For the pre-trained model design corresponding to the code data, a classic Decoder-based framework is chosen. Building upon this, this invention further pre-trains the model on the 1.3 billion parameter scale of the DeepSeekCoder-Base model weights. This allows the model to benefit from the knowledge and representations learned by DeepSeekCoder-Base, which is particularly advantageous in code-related tasks.

[0050] 2) Input / output representation

[0051] The formal description of the input received by the germination-based pre-training model proposed in this invention is as follows, where w i [x] represents the i-th token of the original data x'. [BOS] and [EOS] are special markers in the input, representing the start and end of the sentence.

[0052] x'=[BOS],w1,w2,…,w m [EOS]

[0053] Based on the logical structure representation in the data, the implied logical structure can be serialized as follows:

[0054] x = [BOS], g1, g2, ..., g k [EOS]

[0055] Among them, g i This represents a token after the logical structure has been serialized, such as a specific grammar rule in code. The model generates structured code by inputting and outputting different grammar rules. If the logically structured data itself already has a highly logically structured sequence representation, then x and x' can also have the same form. For ease of demonstration, x will be consistently used as the input data form of the model in the following text.

[0056] Accordingly, the output can be formally described as follows, where y i [i] represents the token of the i-th output, and [BOS] and [EOS] represent the start and end markers in the output, respectively.

[0057] y = [BOS], y1, y2, ..., y z [EOS]

[0058] Finally, after completing all the necessary growth steps, the model transforms the output back into the original data form. Similarly, if the output data is already highly structured, it can be directly generated on the original data structure without any structured representation. Therefore, no transformation is needed during input or output.

[0059] It is worth noting that, depending on the different requirements of different tasks, the input x can be natural language descriptive text (such as the content contained in logically structured code in code generation tasks) or code that needs to be modified (such as old logically structured code data in code enhancement tasks).

[0060] 3) Pre-training task

[0061] The pre-training task involved in this invention aims to effectively combine process data pairs grown from different locations based on the same data point. Simultaneously, to preserve the capabilities and adaptability of the pre-trained model on traditional tasks, the germination strategy proposed in this invention can also be combined with traditional pre-training tasks.

[0062] a) Pre-training task based on germination strategy

[0063] This pre-training task aims to fully explore the internal logic and structure of the data using a germination strategy. Unlike traditional pre-training tasks that add random masks or noise to the input data and then predict sequentially, this invention focuses on the logical structure of the data, starting growth at any position within the structured data until a data form with a complete logical structure emerges. It is noteworthy that the germination and growth designed in this invention can start from any position, and the steps of each growth can be adjusted according to different tasks.

[0064] In a formal sense, the loss of this mission can be described as follows:

[0065]

[0066] Where t represents the current generation step, y t Let represent the content of the t-th node in the generated logical structure that needs to be predicted, and z represent the content of z tokens that need to be predicted in this growth process. pred This represents the unknown data that needs to be predicted and is already marked in the logical structure. `i` represents the data generated during the `i`th growth process; a total of `n` growth processes can generate complete logically structured data. `P` (θ) This represents the probability distribution fitted by the code pre-trained model.

[0067] b) Tasks based on traditional pre-training

[0068] In order to make the final model compatible with traditional model usage, the second pre-training of this invention is based on the next token prediction task of traditional pre-training, specifically generating the content of the token step by step from front to back.

[0069] In a formal sense, the loss of this mission can be described as follows:

[0070]

[0071] Here, x contains a total of k tokens. The pre-training task predicts the content of the t-th token by taking all the tokens before the t-th token as input, until all tokens are generated.

[0072] c) Combination of pre-training objectives

[0073] Each pre-training task can be combined according to its different importance under different tasks, with α and β representing different weights. In the germination pre-training strategy, α is a number greater than 0 and less than or equal to 1, while β is a number greater than or equal to 0 and less than or equal to 1. For clearer demonstration, this invention treats each pre-training task equally and adds the loss values ​​of each task to obtain the overall loss, i.e., both α and β are 0.5. The final loss function is described as follows:

[0074]

[0075] It is worth noting that the present invention retains the germination training strategy proposed in the present invention under different task conditions, and any training strategy added on the basis of this germination strategy is within the scope of the present invention.

[0076] Application Phase: Logically structured data (such as code), requirement descriptions, or logically structured data requiring editing are input into the trained germination-style pre-trained model to generate the desired target logically structured data. Specifically, a strategy for selecting nodes to be grown is first set, such as randomly selecting nodes, generating them gradually from left to right, or selecting the node with the highest predicted probability by the model. Then, based on the selected nodes to be grown, the model is invoked and the germination process is executed. Finally, by repeating the previous step, the logical structure of the target data is gradually improved until data with a complete logical structure is generated.

[0077] III. Evaluation of the Technical Effects of the Invention

[0078] Finally, in order to evaluate the functionality of the present invention, the present invention selected code data generation with logical structure characteristics to evaluate the ability of the germination pre-trained model, specifically the code structure filling task.

[0079] Code Structure Completion Task: This task involves inputting several code snippets with missing structures. A pre-trained model automatically fills in the missing structural content at a specific location according to instructions. This task requires the model to understand the locations of missing structures and automatically generate content that combines the logical dependencies between them. This task is used to evaluate the invention's ability to understand the internal logical structure of logical data and fill in the corresponding correct structural content.

[0080] Evaluation results show that in the code structure completion task, this invention achieves a multi-token prediction accuracy exceeding 20% ​​on a dataset of 500 test cases. It's worth noting that, under the metric of perfectly correct multi-token prediction, if even one token is incorrectly generated, it is considered an erroneous result. In contrast, models with the same parameter size often achieve accuracy below 5% because they struggle to fill in structural information that conforms to the preceding and following logical constraints.

[0081] Practical application scenarios of this invention:

[0082] Logically structured data plays a crucial role in scientific computing, knowledge representation, automated design, and data management due to its high precision and ease of parsing. Automatically generating this data can effectively reduce the manual construction costs associated with it. For example, automatically generating logically structured code data can significantly improve software development efficiency and reduce labor costs in the face of increasing software complexity. Pre-trained models have great potential for automatically generating logically structured data. Utilizing the germination-based pre-training strategy of this invention, pre-trained models can further understand and utilize the internal logical and structural information of such data, thereby further improving efficiency.

[0083] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and to implement it accordingly. Those skilled in the art will understand that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the content disclosed in the embodiments of this specification; the scope of protection of the present invention is defined by the claims.

Claims

1. A germinal pre-training method for modeling logically structured data, characterized in that, The method includes: Perform data logic analysis on the raw data to obtain a structured representation of the raw data; Data generation is performed based on the structured representation to obtain logically structured data. And structure the data according to this logic. The growth path during data germination involves acquiring several data sample pairs. ; wherein, the logically structured data Data that has a clearly defined logical rule or semantic structure, and that is logically structured data. For code, Representing data In the Data before germination during the second germination process Representing data In the The data germination process is defined as the process by which data grows from zero to gradually develop into final data with a complete logical structure. Each germination can begin from any position where further growth is needed. This logically structured data... In the case of code, the code content grows from the root node of the abstract syntax tree until it grows into complete code content. The data germination process can choose to germinate from any node of the abstract syntax tree. The result of the growth can restore the complete code content under the constraints of the code logic structure. Based on the data sample pairs Complete the training of the target model to obtain the germination pre-trained model; Among them, based on the data sample pairs Complete the pre-training of the target model to obtain the pre-trained target model, including: The target model is pre-trained on DeepSeekCoder-Base; wherein the target model is built on the Decoder framework. Data samples Input into the pre-trained target model and combine with data samples Obtain the loss of the pre-training task based on the germination strategy. And the loss of the next token prediction task The pre-training task based on the germination strategy refers to taking the logical structure representation of data as the object, starting to grow at any position in the structured data until a data form with a complete logical structure is germinated. The germination strategy includes: randomly selecting a node as the node to be grown, gradually generating from left to right, and selecting the node to be grown according to probability. Combined with the aforementioned loss and loss The final loss was obtained; among them, Represents the parameters of the target model; Backpropagation is performed based on the final loss to update the parameters of the pre-trained target model.

2. The method according to claim 1, characterized in that, The loss ;in, Represents logically structured data Number of germinations, Indicates the first Prediction is needed during the secondary germination process. The content of each token This represents the probability distribution fitted by the target model. Represents data samples The content of the t-th node in the data. Represents data samples The unknown data that needs to be predicted.

3. The method according to claim 1, characterized in that, The loss ;in, Represents data samples The number of tokens in the data. This represents the probability distribution fitted by the target model. Represents data samples The first in One token, Represents data samples The first in The token to the 1st A token.

4. The method according to claim 1, characterized in that, The data sample pair After completing the training of the target model and obtaining the germination pre-trained model, the following steps are also included: A sprouting strategy is set, which includes: randomly selecting a node as a node to be grown, gradually generating nodes from left to right, and selecting nodes to be grown based on probability. Select data to be grown based on the germination strategy. The nodes to be grown; Based on the selected nodes to be grown, the germination-based pre-trained model is invoked to obtain the data to be grown. And based on the data to be grown Re-execute the selection of data to be grown based on the germination strategy. The nodes to be grown will be selected until the set number of germinations is reached.

5. A germinal pre-training system for logically structured data modeling, characterized in that, The system includes: The data transformation module is used to perform data logic analysis on the raw data and obtain a structured representation of the raw data; The data germination module is used to germinate data based on the structured representation to obtain logically structured data. And structure the data according to this logic. The growth path during data germination involves acquiring several data sample pairs. ; wherein, the logically structured data Data that has a clearly defined logical rule or semantic structure, and that is logically structured data. For code, Representing data In the Data before germination during the second germination process Representing data In the The data germination process is defined as the process by which data grows from zero to gradually develop into final data with a complete logical structure. Each germination can begin from any position where further growth is needed. This logically structured data... In the case of code, the code content grows from the root node of the abstract syntax tree until it grows into complete code content. The data germination process can choose to germinate from any node of the abstract syntax tree. The result of the growth can restore the complete code content under the constraints of the code logic structure. The model training module is used to train pairs of data samples. Complete the training of the target model to obtain the germination pre-trained model; Among them, based on the data sample pairs Complete the pre-training of the target model to obtain the pre-trained target model, including: The target model is pre-trained on DeepSeekCoder-Base; wherein the target model is built on the Decoder framework. Data samples Input into the pre-trained target model and combine with data samples Obtain the loss of the pre-training task based on the germination strategy. And the loss of the next token prediction task The pre-training task based on the germination strategy refers to taking the logical structure representation of data as the object, starting to grow at any position in the structured data until a data form with a complete logical structure is germinated. The germination strategy includes: randomly selecting a node as the node to be grown, gradually generating from left to right, and selecting the node to be grown according to probability. Combined with the aforementioned loss and loss The final loss was obtained; among them, Represents the parameters of the target model; Backpropagation is performed based on the final loss to update the parameters of the pre-trained target model.

6. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the germination pre-training method for logically structured data modeling as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the germination pre-training method for logically structured data modeling as described in any one of claims 1-4.

8. A computer program product, characterized in that, When the computer program product is run on a computer device, the computer device performs the germination pre-training method for logically structured data modeling as described in any one of claims 1-4.

Images