Code generation method and device based on up-multiplexing mixed diversity expert large model

Abstract

The invention provides a code generation method and device based on an up-multiplexing mixed diversity expert large model, and belongs to the field of artificial intelligence, and the method comprises the steps: obtaining code description information; inputting the code description information into a trained code large model, and obtaining a code output by the code large model; the code large model is a large model based on an upper multiplexing mixed diversity expert; the large code model is constructed by adding a shared expert and a plurality of common experts on the basis of the dense model; sharing parameters of an FFN module of the expert multiplexing dense model; the shared experts are used for extracting general knowledge of the lexical elements; a common expert adopts a random initialization mode to initialize parameters; and common experts are used for extracting special knowledge of lexical elements. According to the method, a shared expert and a plurality of common experts are added on the basis of a dense model to construct a code large model, and a diversity multiplexing mechanism is adopted, so that the accuracy of generated codes is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a code generation method and device based on a multiplexed mixed diversity expert large model. Background Art

[0002] Large Language Model (LLM) refers to a deep learning model that can handle a variety of natural language tasks and has a large number of parameters. Large Code Model refers to a large language model used for code generation, and enhanced generation of large code models aims to further design the pre-trained large code model to improve the quality of the code generated by the model.

[0003] Among the code generation technologies for large models of related codes, LLM based on the Mixture of Experts (MoE) architecture is one of the hot research directions in the LLM field. Currently, LLM based on the MoE architecture can achieve better results with fewer activation parameters. Usually, the sparse LLM of the MoE architecture is similar to the dense LLM and needs to be trained from scratch. In the latest solution, the upcycling method is used to copy the feedforwad network of the original dense model to obtain multiple experts, and a gated routing mechanism is added. The other components of the model remain unchanged from the original dense model, and training is performed on this basis.

[0004] The large code model using the above technical solution partially reuses the sedimentation cost of dense model training, which can save computing power compared to training from scratch. However, the accuracy of the code generated by this large code model is low. Summary of the invention

[0005] The present invention provides a code generation method and device based on a multiplexed mixed diversity expert large model, which are used to solve the technical problem of low code accuracy generated by a large code model in the prior art.

[0006] In a first aspect, the present invention provides a code generation method based on a multiplexed mixed diversity expert large model, comprising: Get code description information; Input the code description information into a trained code model to obtain the code output by the code model; The code big model is a big model based on the reused mixed diversity experts; the code big model is constructed by adding a shared expert and M ordinary experts on the basis of the dense model; the shared expert reuses the parameters of the FFN module of the dense model; the shared expert is used to extract the general knowledge of word units; the ordinary experts use random initialization to initialize parameters; the ordinary experts are used to extract the special knowledge of word units, where M is an integer greater than or equal to 2.

[0007] In some embodiments, the code macromodel further includes a routing network and a summing module; The routing network is used to send each word to be processed to the corresponding N general experts; N is an integer greater than or equal to 2, and N is less than or equal to M; The summing module is used to sum the word-grams output by the N ordinary experts and the word-grams output by the shared expert.

[0008] In some embodiments, summing the word-grams output by the N ordinary experts and the word-grams output by the shared expert comprises: Performing weighted summation on the word-units output by the N ordinary experts to obtain a summation result; The sum result and the word-unit output by the shared expert are summed.

[0009] In some embodiments, the routing network is used to determine a score for each general expert based on a weight matrix of each general expert, and to determine the N general experts with the highest scores for processing the word-gram.

[0010] In some embodiments, the method further comprises: determining differences in word unit outputs of shared experts and differences in word unit outputs of common experts; Determining a diversity loss function value based on the word unit output difference of the shared expert and the word unit output difference of the common expert; Determining a total loss based on the diversity loss function value; Training of the code model is guided based on the total loss.

[0011] In some embodiments, the dense model is a DeepSeek-Coder-Base 1.3B model.

[0012] In a second aspect, the present invention further provides a code generation device based on a multiplexed mixed diversity expert large model, comprising: The acquisition module is used to obtain code description information; A generation module, used for inputting the code description information into a trained code model to obtain the code output by the code model; The code big model is a big model based on the reused mixed diversity experts; the code big model is constructed by adding a shared expert and M ordinary experts on the basis of the dense model; the shared expert reuses the parameters of the FFN module of the dense model; the shared expert is used to extract the general knowledge of word units; the ordinary experts use random initialization to initialize parameters; the ordinary experts are used to extract the special knowledge of word units, where M is an integer greater than or equal to 2.

[0013] In the third aspect, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the code generation method based on the multiplexed mixed diversity expert large model as described above is implemented.

[0014] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a code generation method based on a multiplexed mixed diversity expert large model as described in any one of the above.

[0015] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements a code generation method based on a multiplexed mixed diversity expert large model as described in any one of the above.

[0016] The code generation method and device based on the multiplexed mixed diversity expert large model provided by the present invention add a shared expert and multiple ordinary experts to construct the code large model on the basis of the dense model, adopt the diversity multiplexing mechanism, increase the differences between experts, reduce the redundancy between the information extracted by different experts, and improve the accuracy of the generated code. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flowchart of the code generation method based on the multiplexed mixed diversity expert large model provided by the present invention.

[0019] Figure 2 It is a schematic diagram of the architecture of a large model based on multiplexing mixed diversity experts provided by the present invention.

[0020] Figure 3It is a structural schematic diagram of a code generation device based on a multiplexed mixed diversity expert large model provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] Figure 1 : is a flow chart of a code generation method based on a multiplexed mixed diversity expert large model provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 101: Obtain code description information.

[0024] Specifically, in the embodiment of the present application, to generate code using a large code model, it is first necessary to obtain code description information.

[0025] The code description information includes at least one of function description information, syntax description information, personalized coding style requirements, etc.

[0026] The code description may be a natural language description.

[0027] Step 102, input the code description information into the trained code big model, and obtain the code output by the code big model; the code big model is a big model based on the reused mixed diversity experts (Mixture of DiverseExperts, MoDE); the code big model is constructed by adding a shared expert and M ordinary experts on the basis of a dense model; the shared expert reuses the parameters of the FFN module of the dense model; the shared expert is used to extract the general knowledge of word units; the ordinary expert uses random initialization to initialize parameters; the ordinary expert is used to extract the special knowledge of word units, where M is an integer greater than or equal to 2.

[0028] Specifically, in the embodiment of the present application, after obtaining the code description information, it is necessary to use a large code model to understand the problem described in natural language, and accordingly write a computer program that meets the requirements.

[0029] Figure 2Schematic diagram of the architecture of the large model based on multiplexing mixed diversity experts provided by the present invention, such as Figure 2 As shown, the code big model in the embodiment of the present application is a big model based on the reused mixed diversity experts. The code big model is constructed by adding a shared expert and M ordinary experts on the basis of the dense model; the shared expert reuses the parameters of the FFN module of the dense model; the shared expert is used to extract the general knowledge of the word unit; the ordinary expert uses random initialization to initialize parameters; the ordinary expert is used to extract the special knowledge of the word unit, where M is an integer greater than or equal to 2.

[0030] The up-reuse hybrid expert diversity initialization strategy disclosed in the present invention is a strategy designed to improve the diversity among experts, that is, a diversity up-reuse mechanism, which uses a copy method to reuse the parameters of the FFN module of the original dense model for shared experts, and uses a random initialization method to initialize other ordinary experts to increase diversity. This method is different from the initialization method of the original up-reuse hybrid expert large model. While reusing the capabilities of the dense model, this method avoids the problem of completely identical experts after initialization, increases the differences between experts, reduces the redundancy between information extracted by different experts, and improves the accuracy of the generated code.

[0031] In some embodiments, the code macromodel further includes a routing network and a summing module; The routing network is used to send each word to be processed to the corresponding N general experts; N is an integer greater than or equal to 2, and N is less than or equal to M; The summing module is used to sum the word-grams output by the N ordinary experts and the word-grams output by the shared expert.

[0032] Specifically, if Figure 2 As shown, in the embodiment of the present application, the code model also includes a routing network (Router) and a summation module.

[0033] The routing network is used to send each word to be processed to the corresponding N general experts; N is an integer greater than or equal to 2, and N is less than or equal to M.

[0034] The summing module is used to sum the word-grams output by the N ordinary experts and the word-grams output by the shared expert.

[0035] In some embodiments, summing the word-grams output by the N ordinary experts and the word-grams output by the shared expert comprises: Performing weighted summation on the word-units output by the N ordinary experts to obtain a summation result; The sum result and the word-unit output by the shared expert are summed.

[0036] Specifically, if Figure 2 As shown, in the embodiment of the present application, the summation module includes a weighted summation (Weighted Sum) unit and a summation (Sum) unit.

[0037] The weighted summing unit performs weighted summing on the word-units output by the N ordinary experts to obtain a summing result.

[0038] The summing unit sums the summing result and the word unit output by the shared expert.

[0039] In some embodiments, the routing network is used to determine a score for each general expert based on a weight matrix of each general expert, and to determine the N general experts with the highest scores for processing the word-gram.

[0040] Specifically, in the embodiment of the present application, the routing network is used to determine the score of each ordinary expert based on the weight matrix of each ordinary expert. The calculation formula is as follows:

[0041] in, represents the score of ordinary experts, represents the weight matrix of ordinary experts, Indicates the word to be processed. Represents the bias term.

[0042] The selected top-N common expert index sets are as follows:

[0043] The output of the sum module is as follows:

[0044] in, Represents the output result of the summation module, represents the output of the shared expert, represents the weight value corresponding to the jth ordinary expert, Represents the output result of the jth ordinary expert.

[0045] The code generation method based on the multiplexed mixed diversity expert large model provided by the present invention adds a shared expert and multiple ordinary experts to construct a large code model on the basis of the dense model, adopts a diversity multiplexing mechanism, increases the differences between experts, reduces the redundancy between the information extracted by different experts, and improves the accuracy of the generated code.

[0046] In some embodiments, the method further comprises: determining differences in word unit outputs of shared experts and differences in word unit outputs of common experts; Determining a diversity loss function value based on the word unit output difference of the shared expert and the word unit output difference of the common expert; Determining a total loss based on the diversity loss function value; Training of the code model is guided based on the total loss.

[0047] Specifically, in an embodiment of the present application, during the training of the large code model, the training of the large code model is guided based on the diversity loss.

[0048] First, the word unit output difference of the shared experts is calculated. The calculation formula is as follows:

[0049] in, Represents the output difference after the word is input to the shared expert, Representing the Shared Expert word The output after that, Representing the Shared Expert j word The output after that, t Represents the number of input tokens.

[0050] Then calculate the word unit output difference of ordinary experts, the calculation formula is as follows:

[0051] in, Representative k The difference in word output between ordinary experts, represents the difference in word output of all common experts, Representative k General experts handle word The output after that, Representative k General experts handle j word The output after that, Represents the total number of experts, n Represents the number of input tokens.

[0052] Then calculate the diversity loss function value , the specific calculation method is as follows:

[0053] In the embodiment of the present application, the above loss function can be set Compare Large, that is, it encourages shared experts to have smaller differences and tend to extract common knowledge, while the differences between ordinary experts are larger and tend to extract more domain or task knowledge.

[0054] Then the diversity loss is added as an additional loss to the overall loss function of the MoE large model. Usually the loss function of the MoE structure large model is It consists of two parts, as follows:

[0055] That is, the loss function of the large model Balance loss between experts , the above diversity loss Add to the final total loss , the calculation formula is as follows:

[0056] In the actual training process, α is set to 0.01 and β is set to 0.0001.

[0057] The multiplexed hybrid expert diversity loss function in the embodiment of the present application is an additional loss function designed to increase the diversity among experts. The loss function guides the model to output smaller words after the shared experts process than the words after other ordinary experts process, because the shared experts are more inclined to extract general knowledge, while ordinary experts may be more inclined to extract specialized knowledge in their respective fields or tasks. This further increases the differences between experts while guiding the experts to complete the division of labor in knowledge extraction, and further improves the accuracy of the generated code.

[0058] The above method is further explained below with a specific example: In this example, the DeepSeek-Coder-Base 1.3B model is used as the dense model, that is, the DeepSeek-Coder-Base 1.3B model is used as the basis of the code large model. On the basis of the DeepSeek-Coder-Base 1.3B model, a shared expert and 7 ordinary experts are added to construct the code large model. N is set to 5, that is, the output results of the 5 ordinary experts with the highest scores are selected.

[0059] In this example, experiments are conducted on the code evaluation datasets HumalEval and HumanEval+. The accuracy comparison of the codes output by different models is shown in Table 1.

[0060] Table 1 Accuracy of codes output by different models

[0061] In the table, DeepSeek-Coder-Base 1.3B is an open source 1B code LLM, SU-MoE represents an LLM using a multiplexed mixed expert architecture, and MoDE is a large model based on multiplexed mixed diversity experts in this application.

[0062] By comparison, the proposed method can achieve relative performance improvements of more than 2.7% and 3.5% on HumalEval and HumanEval+, proving its effectiveness in code generation tasks.

[0063] In addition, while improving the model performance, the method of the present invention does not significantly increase the burden of model training and does not affect the convergence speed of model training.

[0064] In the embodiment of the present application, during the training process of the code large model, evol-codealpaca-v1 is selected as the code generation task training set, and then the code large model is trained and fine-tuned. The main training parameter configurations are shown in Table 2.

[0065] Table 2 Training parameter configuration table

[0066] The following describes the code generation device based on the up-multiplexed mixed diversity expert big model provided by the present invention. The code generation device based on the up-multiplexed mixed diversity expert big model described below and the code generation method based on the up-multiplexed mixed diversity expert big model described above can be referenced to each other.

[0067] Figure 3 Schematic diagram of the structure of the code generation device based on the multiplexed mixed diversity expert large model provided by the present invention, such as Figure 3 As shown, the code generation device based on the multiplexed mixed diversity expert large model provided by the present invention includes: The acquisition module 301 is used to acquire code description information; The generation module 302 is used to input the code description information into the trained code model to obtain the code output by the code model; The code big model is a big model based on the reused mixed diversity experts; the code big model is constructed by adding a shared expert and M ordinary experts on the basis of the dense model; the shared expert reuses the parameters of the FFN module of the dense model; the shared expert is used to extract the general knowledge of word units; the ordinary experts use random initialization to initialize parameters; the ordinary experts are used to extract the special knowledge of word units, where M is an integer greater than or equal to 2.

[0068] Specifically, the code generation device based on the multiplexed mixed diversity expert big model provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned code generation method embodiment based on the multiplexed mixed diversity expert big model, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those of the method embodiment will not be described in detail here.

[0069] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the code generation method based on the multiplexed mixed diversity expert large model, the method comprising: Get code description information; Input the code description information into a trained code model to obtain the code output by the code model; The code big model is a big model based on the reused mixed diversity experts; the code big model is constructed by adding a shared expert and M ordinary experts on the basis of the dense model; the shared expert reuses the parameters of the FFN module of the dense model; the shared expert is used to extract the general knowledge of word units; the ordinary experts use random initialization to initialize parameters; the ordinary experts are used to extract the special knowledge of word units, where M is an integer greater than or equal to 2.

[0070] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0071] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the code generation method based on the multiplexed mixed diversity expert large model provided by the above methods, the method comprising: Get code description information; Input the code description information into a trained code model to obtain the code output by the code model; The code big model is a big model based on the reused mixed diversity experts; the code big model is constructed by adding a shared expert and M ordinary experts on the basis of the dense model; the shared expert reuses the parameters of the FFN module of the dense model; the shared expert is used to extract the general knowledge of word units; the ordinary experts use random initialization to initialize parameters; the ordinary experts are used to extract the special knowledge of word units, where M is an integer greater than or equal to 2.

[0072] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the code generation method based on the multiplexed mixed diversity expert large model provided by the above methods, the method comprising: Get code description information; Input the code description information into a trained code model to obtain the code output by the code model; The code big model is a big model based on the reused mixed diversity experts; the code big model is constructed by adding a shared expert and M ordinary experts on the basis of the dense model; the shared expert reuses the parameters of the FFN module of the dense model; the shared expert is used to extract the general knowledge of word units; the ordinary experts use random initialization to initialize parameters; the ordinary experts are used to extract the special knowledge of word units, where M is an integer greater than or equal to 2.

[0073] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0074] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0075] It should also be noted that: "Determine B based on A" in the embodiments of the present application means that the factor A should be considered when determining B. It is not limited to "B can be determined based on A alone", but should also include: "Determine B based on A and C", "Determine B based on A, C and E", "Determine C based on A, and further determine B based on C", etc. It can also include taking A as a condition for determining B, for example, "When A meets the first condition, use the first method to determine B"; for example, "When A meets the second condition, determine B", etc.; for example, "When A meets the third condition, determine B based on the first parameter", etc. Of course, it can also be a condition that takes A as a factor for determining B, for example, "When A meets the first condition, use the first method to determine C, and further determine B based on C", etc.

[0076] In the embodiments of the present application, the term "plurality" refers to two or more than two, and other quantifiers are similar.

[0077] In the embodiments of the present application, the terms "first", "second", etc. are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first" and "second" are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A code generation method based on multiplexing mixed diversity expert large model, characterized in that: include: Get code description information; Input the code description information into a trained code model to obtain the code output by the code model; The code big model is a big model based on multiplexing mixed diversity experts; The code big model is constructed by adding a shared expert and M ordinary experts on the basis of the dense model; the shared expert reuses the parameters of the FFN module of the dense model; the shared expert is used to extract the general knowledge of word units; the ordinary experts use random initialization to initialize parameters; the ordinary experts are used to extract the special knowledge of word units, where M is an integer greater than or equal to 2.

2. The code generation method based on multiplexing mixed diversity expert large model according to claim 1 is characterized in that: The code big model also includes a routing network and a summing module; The routing network is used to send each word to be processed to the corresponding N general experts; N is an integer greater than or equal to 2, and N is less than or equal to M; The summing module is used to sum the word-grams output by the N ordinary experts and the word-grams output by the shared expert.

3. The code generation method based on multiplexing mixed diversity expert large model according to claim 2 is characterized in that: Summing the word-grams output by the N ordinary experts and the word-grams output by the shared expert comprises: Performing weighted summation on the word-units output by the N ordinary experts to obtain a summation result; The sum result and the word-unit output by the shared expert are summed.

4. The code generation method based on multiplexing mixed diversity expert large model according to claim 2 is characterized in that: The routing network is used to determine the score of each general expert based on the weight matrix of each general expert, and determine the N general experts with the highest scores to process the word-unit.

5. The code generation method based on multiplexing mixed diversity expert large model according to claim 1 is characterized in that: The method further comprises: determining differences in word unit outputs of shared experts and differences in word unit outputs of common experts; Determining a diversity loss function value based on the word unit output difference of the shared expert and the word unit output difference of the common expert; Determining a total loss based on the diversity loss function value; Training of the code model is guided based on the total loss.

6. The code generation method based on multiplexing mixed diversity expert large model according to any one of claims 1 to 5 is characterized in that: The dense model is the DeepSeek-Coder-Base 1.3B model.

7. A code generation device based on multiplexing mixed diversity expert large model, characterized in that: include: The acquisition module is used to obtain code description information; A generation module, used for inputting the code description information into a trained code model to obtain the code output by the code model; The code big model is a big model based on multiplexing mixed diversity experts; The code big model is constructed by adding a shared expert and M ordinary experts on the basis of the dense model; the shared expert reuses the parameters of the FFN module of the dense model; the shared expert is used to extract the general knowledge of word units; the ordinary experts use random initialization to initialize parameters; the ordinary experts are used to extract the special knowledge of word units, where M is an integer greater than or equal to 2.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, it implements the code generation method based on the multiplexed mixed diversity expert large model as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the code generation method based on the multiplexed mixed diversity expert large model as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the code generation method based on the multiplexed mixed diversity expert large model as described in any one of claims 1 to 6 is implemented.

Images