A programming-based large language model fine-tuning-free pre-training method and device

Abstract

The application provides a programming-based large language model fine-tuning-free pre-training method and device, which comprises the following steps: extracting keywords of each sample in a code prompt training set to generate a keyword set corresponding to the code prompt training set; assigning a compression symbol to each keyword in the keyword set; replacing each keyword in the code prompt training set with its compression symbol to obtain a first code prompt training set; and pre-training a general large language model in a prompt learning mode based on the first code prompt training set. The application takes advantage of the fact that a program compiler is not sensitive to the semantics of keywords, and symbolically compresses and encodes the keywords in the code prompt training set, thereby greatly reducing the length of the code prompt information input to the large language model, so that the code prompt can be input as few times as possible and as much as possible in the case that the large language model has a length limit for dialogue input, and the efficient pre-training of the large language model is realized.

Description

Technical Field

[0001] This invention relates to the field of computer model training technology, and in particular to a method and apparatus for pre-training large language models without fine-tuning based on programming. Background Technology

[0002] Currently, generative pre-trained large language models, exemplified by OpenAI's GPT, are rapidly developing. They possess powerful dialogue generation capabilities and can solve a large number of knowledge-based problems through question-and-answer responses. This pre-trained large language model is derived by fine-tuning a general-purpose large language model using domain-specific knowledge, further optimizing the model's answer quality for that specific domain.

[0003] However, fine-tuning a general-purpose large language model for specific domains faces the following challenges:

[0004] 1. It is necessary to construct a substantial and effective training set in advance, and repeatedly verify the results of the fine-tuning of the basic model to avoid regression in generation capabilities.

[0005] 2. If domain-specific knowledge changes, the training should primarily focus on the changed knowledge; the fine-tuned large language model does not possess this capability.

[0006] 3. After fine-tuning, if the basic model needs to be replaced with a model from another domain, it needs to be fine-tuned again.

[0007] 4. If the training information used for fine-tuning is highly sensitive, once fine-tuning begins, this information will be immediately internalized into the large model, which may be exploited by others and lead to leakage.

[0008] 5. Although large language models have the ability to remember context, they have strict limitations on the length of input and cannot directly input large training sets through dialogue.

[0009] Considering the aforementioned challenges and the fact that general-purpose large language models possess certain code generation capabilities, a fine-tuning-free pre-training method for large language models has been proposed, utilizing domain-specific code hint texts for prompt learning. This method enables the model to generate question-and-answer code for specific domains. However, machines have limitations on the amount of input they can process at one time, while the training set of code hint texts is relatively large, which will affect the model's effectiveness in learning domain-specific knowledge.

[0010] Therefore, a new method for fine-tuning-free pre-training of large language models is needed. Summary of the Invention

[0011] To address the aforementioned problems, this invention provides a method and apparatus for pre-training a large language model without fine-tuning, based on programming principles. Utilizing the semantic insensitivity of compilers to keywords, the keywords in the code prompt training set are symbolically compressed and encoded, significantly reducing the length of the code prompt information input to the large language model. This allows for the input of code prompts in fewer but more frequent instances, given the length limitations of the large language model's dialogue input, thus achieving efficient pre-training of the large language model. Based on the basic programming capabilities of the large language model, when applying the pre-trained large language model without fine-tuning, the model responds with a reply code based on the input question code that has undergone symbolic compression and encoding. The reply code is then symbolically decompressed and encoded to obtain a human-understandable program. The application process is also simple and easy to implement.

[0012] In a first aspect, the present invention provides a programming-based method for fine-tuning-free pre-training of large language models, the method comprising:

[0013] Extract keywords from each sample in the code hint training set to generate a keyword set corresponding to the code hint training set;

[0014] Assign a compression symbol to each keyword in the keyword set;

[0015] Replace each keyword in the code prompt training set with its compressed symbol to obtain the first code prompt training set;

[0016] Based on the first code prompt training set, a prompt learning method is used to pre-train the general large language model.

[0017] According to the programming-based large language model pre-training method provided by the present invention, the samples in the code prompt training set are presented in a "prompt-complete" manner;

[0018] The prompt is a code description of a question or an example;

[0019] The completion refers to the code description that guides the model to generate the best response.

[0020] According to the programming-based large language model pre-training method provided by the present invention, the extraction of keywords from each sample in the code prompt training set includes:

[0021] Generate an abstract syntax tree for each of the samples;

[0022] Specific type words are extracted from the abstract syntax tree, and the extracted specific type words are used as keywords for each sample.

[0023] According to the programming-based large language model pre-training method provided by the present invention, the specific type of words includes, but is not limited to, variable names, method names, and special symbols.

[0024] According to the programming-based large language model pre-training method without fine-tuning provided by the present invention, when assigning a compressed symbol to each keyword in the keyword set, the following conditions must be met simultaneously:

[0025] Condition 1: Different keywords in the keyword set have different compression symbols;

[0026] Condition 2: When replacing each keyword in the code prompt training set with its compressed symbol, the semantics of the code must not be changed.

[0027] The method for pre-training a large language model without fine-tuning based on programming, provided by the present invention, further includes, after assigning a compressed symbol to each keyword in the keyword set:

[0028] A mapping table is established based on the mapping relationship between each keyword and its compressed symbol in the keyword set.

[0029] According to the programming-based large language model pre-training method provided by the present invention, the application process of the pre-trained large language model includes:

[0030] Get the code representation of the question to be asked;

[0031] The code representation is input into the pre-trained large language model to obtain the first response;

[0032] Based on the mapping table, the compressed symbols in the first response are replaced with their keywords to obtain the answer to the question.

[0033] Secondly, the present invention provides a programming-based large language model pre-training device without fine-tuning, the device comprising:

[0034] The generation module is used to extract keywords from each sample in the code hint training set to generate a keyword set corresponding to the code hint training set;

[0035] The allocation module is used to assign a compression symbol to each keyword in the keyword set;

[0036] The replacement module is used to replace each keyword in the code prompt training set with its compressed symbol to obtain the first code prompt training set.

[0037] The pre-training module is used to pre-train the general large language model based on the first code prompt training set and using a prompt learning method.

[0038] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the programming-based large language model fine-tuning-free pre-training method as described in the first aspect.

[0039] Fourthly, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the programming-based large language model fine-tuning-free pre-training method as described in the first aspect.

[0040] This invention provides a method and apparatus for pre-training a large language model without fine-tuning based on programming. The method includes: extracting keywords from each sample in a code hint training set to generate a keyword set corresponding to the code hint training set; assigning a compression symbol to each keyword in the keyword set; replacing each keyword in the code hint training set with its compression symbol to obtain a first code hint training set; and pre-training a general-purpose large language model based on the first code hint training set using a hint learning approach. This invention leverages the semantic insensitivity of compilers to keywords by symbolically compressing and encoding the keywords in the code hint training set, thereby significantly reducing the length of the code hint information input to the large language model. This allows for the input of code hints in fewer but more frequent times, given the length limitations of the large language model's dialogue input, thus achieving efficient pre-training of the large language model. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 This is a flowchart illustrating the programming-based large language model pre-training method without fine-tuning provided by the present invention.

[0043] Figure 2 This is a schematic diagram illustrating the pre-training and application process of the large language model provided by this invention;

[0044] Figure 3 This is a schematic diagram of the structure of the programming-based large language model pre-training device without fine-tuning provided by the present invention;

[0045] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention;

[0046] Figure label:

[0047] 410: Processor; 420: Communication interface; 430: Memory; 440: Communication bus. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0049] The following is combined Figures 1-4 The present invention describes a programmable large language model pre-training method and apparatus without fine-tuning.

[0050] Firstly, large language models have length limitations for dialogue input, and the code prompts used for pre-training large language models can easily exceed these limits, making input impossible. Therefore, this invention provides a programmatically based, fine-tuning-free pre-training method for large language models, such as... Figure 1 As shown, the method includes:

[0051] S11: Extract keywords from each sample in the code prompt training set to generate a keyword set corresponding to the code prompt training set;

[0052] As large language models are increasingly integrated into more and more domains, the need for domain-specific knowledge pre-training is becoming more and more urgent. Therefore, the code hint training set mentioned here is constructed by domain experts integrating domain-specific knowledge, and the pre-trained large language model has the ability to generate question-and-answer code for specific domains.

[0053] S12: Assign a compression symbol to each keyword in the keyword set;

[0054] Here, there are no duplicate keywords in the keyword set.

[0055] S13: Replace each keyword in the code prompt training set with its compressed symbol to obtain the first code prompt training set;

[0056] S14: Based on the first code prompt training set, a prompt learning method is used to pre-train the general large language model.

[0057] This invention provides a programming-based method for pre-training large language models without fine-tuning. By utilizing the fact that program compilers are not sensitive to the semantics of keywords, the keywords in the code prompt training set are symbolically compressed and encoded, thereby significantly reducing the length of the code prompt information input to the large language model. This allows for the input of code prompts in as few times as possible, even when the large language model has length limitations for dialogue input, thus achieving efficient pre-training of the large language model.

[0058] Specifically, in the code hint training set in S11, each sample is presented in a "hint-complete" manner;

[0059] The prompt is a code description of a question or an example;

[0060] The completion refers to the code description that guides the model to generate the best response.

[0061] Example: Each piece of training data for a prompt needs to be accompanied by a prompt, which is a description of a problem or example, such as "Java code to calculate the area of ​​a square". In addition, each prompt needs to be accompanied by a completion to guide the large language model on how to generate the best response. The large language model will learn patterns through meta-learning. For the prompt "Java code to calculate the area of ​​a square", a piece of Java code that implements this function can be provided as the completion.

[0062] In other words, by designing a large number of hints based on the characteristics of the domain and using the implementation code of the hints as completion, a code hint training set with domain characteristics can be constructed.

[0063] Specifically, in S11, keywords are extracted from each sample in the code prompt training set, including:

[0064] Generate an abstract syntax tree for each of the samples;

[0065] Specific type words are extracted from the abstract syntax tree, and the extracted specific type words are used as keywords for each sample.

[0066] The specific types of words referred to here include, but are not limited to, variable names, method names, and special symbols.

[0067] Taking JavaScript as an example, we analyze the abstract syntax tree of the input code, traverse the abstract syntax tree, and for each tree node:

[0068] Determine if it is a variable declaration statement or an assignment statement. If it is:

[0069] Add the variable name of this node to the variable table.

[0070] Determine if it is the node that called the function; if it is:

[0071] Add the function name of this node to the variable table.

[0072] This completes the extraction of the long-named set to be replaced.

[0073] In computer science, an Abstract Syntax Tree (AST) is a tree representation of the abstract syntax structure of source code written in a programming language. It consists of data structures called nodes. Each node contains information about its class name. Therefore, by traversing the nodes of the AST for each sample, the keywords for that sample can be extracted.

[0074] Specifically, in step S12, when assigning a compression symbol to each keyword in the keyword set, the following conditions must be met simultaneously:

[0075] Condition 1: Different keywords in the keyword set have different compression symbols;

[0076] Condition 2: When replacing each keyword in the code prompt training set with its compressed symbol, the semantics of the code must not be changed.

[0077] This invention essentially solves the problem of pre-training input for programming code by replacing long-named keywords with short-named compressed symbols.

[0078] This invention first sorts the keywords in a keyword set, and then assigns a compression symbol to each keyword in ascending order of compression symbols. For example, the first keyword uses "a", the second keyword uses "b", the 27th keyword uses "aa", and the 28th keyword uses "ab". However, two conditions must be met during the assignment process: first, different keywords in the keyword set must have different compression symbols; for example, the first and second keywords cannot both be "a" to avoid learning errors; second, when replacing each keyword in the code hint training set with its compression symbol, the semantics of the code must not be altered to ensure the quality of the subsequently generated first code hint training set.

[0079] Specifically, after assigning a compression symbol to each keyword in the keyword set, the process also includes:

[0080] A mapping table is established based on the mapping relationship between each keyword and its compressed symbol in the keyword set.

[0081] In fact, step S13 is equivalent to replacing each keyword in the code prompt training set with its compressed symbol based on the mapping table, to obtain the first code prompt training set.

[0082] This invention proposes a programming-based method for pre-training large language models without fine-tuning, the advantages of which are mainly reflected in the following aspects:

[0083] 1. Reduce the difficulty of fine-tuning. Since there is no need to fine-tune the model, only the pre-training data information needs to be input in the same way as normal dialogue. Therefore, the need for hardware resources for fine-tuning and the complicated process of repeated verification can be avoided.

[0084] 2. It can serve as a supplement to the basic dialogue capabilities of large language models, enabling rapid training of specific domain knowledge for dialogue contexts, which is highly beneficial for pre-training in situations with frequent scene changes.

[0085] 3. Build once, reuse multiple times. This method does not require building the specific training set format required by various vendors' large language base models multiple times. It has strong versatility and can be applied to various large language models.

[0086] 4. Pre-training is performed within the context of each dialogue, and the large language model itself does not leave permanent training data, thus ensuring that data information is not leaked.

[0087] 5. It can effectively solve the problem of large language models' limitations on pre-training input for programming code;

[0088] In summary, the method of this invention has great practicality and promotional value, and is expected to provide an efficient and convenient solution for pre-training work in the field of large language model programming.

[0089] Specifically, the application process of a pre-trained large language model includes:

[0090] Get the code representation of the question to be asked;

[0091] The code representation is input into the pre-trained large language model to obtain the first response;

[0092] Based on the mapping table, the compressed symbols in the first response are replaced with their keywords to obtain the answer to the question.

[0093] This invention is based on a large language model that has basic programming capabilities. When using a large language model that has been pre-trained without fine-tuning, the large language model responds with a response code based on the input question code that has been symbolically compressed and encoded with keywords. By symbolically decompressing and encoding the response code, a human-understandable program can be obtained. The application process is also simple and easy to implement.

[0094] Figure 2 This is a flowchart illustrating the pre-training and application process of a large language model, such as... Figure 2As shown, the core of the pre-training in this invention lies in extracting keywords from the code hint training set based on the Abstract Syntax Tree (AST), automatically generating a mapping table representing the mapping relationship between keywords and their symbolic codes, and replacing the keywords in the code hint training set with their symbolic codes to obtain the first code hint training set. These three steps are referred to as AST extraction, mapping table generation, and mapping table code replacement. In actual pre-training, these three steps can be encapsulated in an encoder. That is, during actual execution, an encoder is connected before the large language model, and the code hint training set is input into the encoder to obtain its output first code hint training set. The first code hint training set is then used to pre-train the large language model.

[0095] In application, a decoder containing a mapping table generated by the encoder is generated. The dialogue question is input into the pre-trained large language model to obtain the original output. The original output is then used to decode the decoder to obtain the dialogue answer.

[0096] Secondly, the present invention describes the programmable large language model pre-training device without fine-tuning provided by the present invention. The programmable large language model pre-training device without fine-tuning described below and the programmable large language model pre-training method without fine-tuning described above can be referred to in correspondence with each other. Figure 3 This is a schematic diagram of the structure of the programming-based large language model pre-training device without fine-tuning provided by the present invention, as shown below. Figure 3 As shown, the device includes:

[0097] The generation module 21 is used to extract keywords from each sample in the code prompt training set to generate a keyword set corresponding to the code prompt training set.

[0098] Allocation module 22 is used to allocate a compression symbol to each keyword in the keyword set;

[0099] Replacement module 23 is used to replace each keyword in the code prompt training set with its compressed symbol to obtain the first code prompt training set;

[0100] The pre-training module 24 is used to pre-train the general large language model based on the first code prompt training set using a prompt learning method.

[0101] This invention provides a programming-based large language model pre-training device that eliminates fine-tuning. By utilizing the fact that the compiler is not sensitive to the semantics of keywords, the keywords in the code prompt training set are symbolically compressed and encoded, thereby significantly reducing the length of the code prompt information input to the large language model. This allows for the input of code prompts in as few times as possible, even when the large language model has length limitations for dialogue input, thus achieving efficient pre-training of the large language model.

[0102] Based on the above embodiments, as an optional embodiment, the samples in the code prompt training set are presented in a "prompt-complete" manner;

[0103] The prompt is a code description of a question or an example;

[0104] The completion refers to the code description that guides the model to generate the best response.

[0105] Based on the above embodiments, as an optional embodiment, the generation module includes:

[0106] A generation unit is used to generate an abstract syntax tree corresponding to each of the samples;

[0107] A setting unit is used to extract specific type words from the abstract syntax tree and use the extracted specific type words as keywords for each sample.

[0108] Based on the above embodiments, as an optional embodiment, the specific type of words include, but are not limited to, variable names, method names, and special symbols.

[0109] Based on the above embodiments, as an optional embodiment, when assigning a compression symbol to each keyword in the keyword set, the following conditions must be met simultaneously:

[0110] Condition 1: Different keywords in the keyword set have different compression symbols;

[0111] Condition 2: When replacing each keyword in the code prompt training set with its compressed symbol, the semantics of the code must not be changed.

[0112] Based on the above embodiments, as an optional embodiment, after assigning a compression symbol to each keyword in the keyword set, the method further includes:

[0113] A mapping table is established based on the mapping relationship between each keyword and its compressed symbol in the keyword set.

[0114] Based on the above embodiments, as an optional embodiment, the application process of the pre-trained large language model includes:

[0115] Get the code representation of the question to be asked;

[0116] The code representation is input into the pre-trained large language model to obtain the first response;

[0117] Based on the mapping table, the compressed symbols in the first response are replaced with their keywords to obtain the answer to the question.

[0118] Thirdly, Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a programming-based large language model fine-tuning-free pre-training method. This method includes: extracting keywords from each sample in the code hint training set to generate a keyword set corresponding to the code hint training set; assigning a compressed symbol to each keyword in the keyword set; replacing each keyword in the code hint training set with its compressed symbol to obtain a first code hint training set; and pre-training a general large language model based on the first code hint training set using a hint learning method.

[0119] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0120] Fourthly, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the programming-based large language model fine-tuning-free pre-training method provided by the above methods. The method includes: extracting keywords from each sample in the code hint training set to generate a keyword set corresponding to the code hint training set; assigning a compressed symbol to each keyword in the keyword set; replacing each keyword in the code hint training set with its compressed symbol to obtain a first code hint training set; and pre-training a general large language model based on the first code hint training set using a hint learning method.

[0121] Fifthly, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the programming-based large language model fine-tuning-free pre-training method provided by the above methods. The method includes: extracting keywords from each sample in the code hint training set to generate a keyword set corresponding to the code hint training set; assigning a compressed symbol to each keyword in the keyword set; replacing each keyword in the code hint training set with its compressed symbol to obtain a first code hint training set; and pre-training a general large language model using a hint learning approach based on the first code hint training set.

[0122] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0123] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A programming-based method for fine-tuning-free pre-training of large language models, characterized in that, The method includes: Extract keywords from each sample in the code hint training set to generate a keyword set corresponding to the code hint training set; Assign a compression symbol to each keyword in the keyword set, and establish a mapping table based on the mapping relationship between each keyword and its compression symbol in the keyword set; Replace each keyword in the code prompt training set with its compressed symbol to obtain the first code prompt training set; Based on the first code prompt training set, a prompt learning method is used to pre-train the general large language model; The application process of a pre-trained large language model includes: Get the code representation of the question to be asked; The code representation is input into the pre-trained large language model to obtain the first response; Based on the mapping table, the compressed symbols in the first response are replaced with their keywords to obtain the answer to the question.

2. The method for pre-training a large language model without fine-tuning based on programming according to claim 1, characterized in that, The samples in the code hint training set are presented in a "hint-complete" manner; The prompt is a code description of a question or an example; The completion refers to the code description that guides the model to generate the best response.

3. The method for pre-training a large language model without fine-tuning based on programming according to claim 1, characterized in that, The extraction of keywords from each sample in the code prompt training set includes: Generate an abstract syntax tree for each of the samples; Specific type words are extracted from the abstract syntax tree, and the extracted specific type words are used as keywords for each sample.

4. The method for pre-training a large language model without fine-tuning based on programming according to claim 3, characterized in that, The specific type of words includes at least variable names, method names, and special symbols.

5. The method for pre-training a large language model without fine-tuning based on programming according to claim 1, characterized in that, When assigning a compression symbol to each keyword in the keyword set, the following conditions must be met simultaneously: Condition 1: Different keywords in the keyword set have different compression symbols; Condition 2: When replacing each keyword in the code prompt training set with its compressed symbol, the semantics of the code must not be changed.

6. A pre-training device for large language models based on programming without fine-tuning, characterized in that, The device includes: The generation module is used to extract keywords from each sample in the code hint training set to generate a keyword set corresponding to the code hint training set. The allocation module is used to assign a compression symbol to each keyword in the keyword set, and to establish a mapping table based on the mapping relationship between each keyword and its compression symbol in the keyword set; The replacement module is used to replace each keyword in the code prompt training set with its compressed symbol to obtain the first code prompt training set. The pre-training module is used to pre-train the general large language model based on the first code prompt training set using a prompt learning method; The application process of a pre-trained large language model includes: Get the code representation of the question to be asked; The code representation is input into the pre-trained large language model to obtain the first response; Based on the mapping table, the compressed symbols in the first response are replaced with their keywords to obtain the answer to the question.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the programmatically based large language model pre-training method without fine-tuning as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the programming-based large language model pre-training method without fine-tuning as described in any one of claims 1 to 5.

Images