Code generation method and device based on superposition of hybrid diversity expert large model

Abstract

The application provides a code generation method and device based on an up-mixing mixed diversity expert large model, and belongs to the field of artificial intelligence. The method comprises the following steps: obtaining code description information; inputting the code description information into a trained code large model to obtain code output by the code large model; the code large model is a large model based on an up-mixing mixed diversity expert; the code large model is constructed by adding a shared expert and multiple ordinary experts on the basis of a dense model; the shared expert reuses parameters of an FFN module of the dense model; the shared expert is used for extracting general knowledge of word elements; the ordinary expert is initialized by using a random initialization method; and the ordinary expert is used for extracting special knowledge of word elements. The code large model is constructed by adding a shared expert and multiple ordinary experts on the basis of a dense model, a diversity up-mixing mechanism is adopted, and the accuracy of generated code is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a code generation method and apparatus based on a large-scale expert model with multiplexed hybrid diversity. Background Technology

[0002] Large Language Model (LLM) refers to a deep learning model that can handle various natural language tasks and has a large number of parameters. Large Code Model refers to a large language model used for code generation, while Augmented Generation of Large Code Models aims to further design the pre-trained Large Code Model to improve the quality of the code generated by the model.

[0003] In code generation techniques for large-scale models, LLM based on the Mixture of Experts (MoE) architecture is one of the hot research directions in the LLM field. Currently, MoE-based LLMs can achieve better results with fewer activation parameters. Typically, sparse LLMs and dense LLMs based on the MoE architecture are similar and require training from scratch. The latest approach uses upcycling to obtain multiple experts by replicating the feedforward network of the original dense model and adding a gating routing mechanism. The other components of the model remain unchanged from the original dense model, and then training is performed on this basis.

[0004] The large code model using the above technical solution partially reuses the accumulated costs of training dense models, which can save computing power compared to training from scratch. However, the code generated by this large code model has low accuracy. Summary of the Invention

[0005] This invention provides a code generation method and apparatus based on a multi-reusable, hybrid, and diverse expert large model, in order to solve the technical problem of low code accuracy in the generation of large code models in the prior art.

[0006] In a first aspect, the present invention provides a code generation method based on a multi-reusable, hybrid, diverse expert large model, comprising:

[0007] Get code description information;

[0008] The code description information is input into the trained code model, and the code output by the code model is obtained.

[0009] The large code model is a large model based on the reuse of hybrid diversity experts; the large code model is constructed by adding one shared expert and M ordinary experts to the dense model; the shared expert reuses the parameters of the FFN module of the dense model; the shared expert is used to extract general knowledge of lexical units; the ordinary experts are initialized with random initialization; the ordinary experts are used to extract special knowledge of lexical units, where M is an integer greater than or equal to 2.

[0010] In some embodiments, the large code model further includes a routing network and a summation module;

[0011] The routing network is used to send each token to be processed to the corresponding N general experts; N is an integer greater than or equal to 2, and N is less than or equal to M;

[0012] The summation module is used to sum the lexical units output by the N ordinary experts and the lexical units output by the shared expert.

[0013] In some embodiments, summing the lexical units output by the N ordinary experts and the lexical units output by the shared expert includes:

[0014] The word units output by the N ordinary experts are weighted and summed to obtain the summation result;

[0015] The summation result and the tokens output by the shared expert are summed.

[0016] In some embodiments, the routing network is used to determine the score of each general expert based on the weight matrix of each general expert, and to determine the N general experts with the highest scores for processing word units.

[0017] In some embodiments, the method further includes:

[0018] Determine the differences in lexical output between shared experts and ordinary experts;

[0019] The diversity loss function value is determined based on the differences in lexical outputs of the shared experts and the differences in lexical outputs of the ordinary experts;

[0020] The total loss is determined based on the value of the diversity loss function.

[0021] The training of the large code model is guided by the total loss.

[0022] In some embodiments, the dense model is the DeepSeek-Coder-Base 1.3B model.

[0023] Secondly, the present invention also provides a code generation device based on a multi-reusable hybrid diversity expert large model, comprising:

[0024] The retrieval module is used to retrieve code description information;

[0025] A generation module is used to input the code description information into a trained code model and obtain the code output by the code model.

[0026] The large code model is a large model based on the reuse of hybrid diversity experts; the large code model is constructed by adding one shared expert and M ordinary experts to the dense model; the shared expert reuses the parameters of the FFN module of the dense model; the shared expert is used to extract general knowledge of lexical units; the ordinary experts are initialized with random initialization; the ordinary experts are used to extract special knowledge of lexical units, where M is an integer greater than or equal to 2.

[0027] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the code generation method based on the multivariate expert big model described above.

[0028] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the code generation method based on the multiplexed hybrid diversity expert large model as described above.

[0029] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the code generation method based on the multivariate expert big model of multiplexing as described above.

[0030] The code generation method and apparatus based on a multi-reuse, hybrid, diverse expert large model provided by this invention adds a shared expert and multiple ordinary experts to a dense model to construct a large code model. By adopting a diversity multi-reuse mechanism, the differences between experts are increased, the redundancy between information extracted by different experts is reduced, and the accuracy of the generated code is improved. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0032] Figure 1This is a flowchart illustrating the code generation method based on a multi-reusable, hybrid diversity expert model provided by the present invention.

[0033] Figure 2 This is a schematic diagram of the architecture of the large model based on the reuse of hybrid diversity experts provided by the present invention.

[0034] Figure 3 This is a schematic diagram of the structure of the code generation device based on the multivariate expert large model of upper reuse provided by the present invention.

[0035] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0037] Figure 1 This is a flowchart illustrating the code generation method based on a multi-reusable, hybrid diversity expert large model provided by the present invention, as shown below. Figure 1 As shown, the method includes the following:

[0038] Step 101: Obtain code description information.

[0039] Specifically, in this embodiment of the application, generating code using a large code model first requires obtaining code description information.

[0040] Code description information includes at least one of the following: functional description information, syntax description information, and personalized coding style requirements.

[0041] The code description can be in natural language.

[0042] Step 102: Input the code description information into the trained code big model and obtain the code output by the code big model; the code big model is a big model based on the reuse of Mixture of Diverse Experts (MoDE); the code big model is constructed by adding one shared expert and M ordinary experts to the dense model; the shared expert reuses the parameters of the FFN module of the dense model; the shared expert is used to extract general knowledge of lexical units; the ordinary experts are initialized with random initialization; the ordinary experts are used to extract specific knowledge of lexical units, where M is an integer greater than or equal to 2.

[0043] Specifically, in this embodiment of the application, after obtaining the code description information, it is necessary to use the large code model to understand the problem described in natural language and write a computer program that meets the requirements accordingly.

[0044] Figure 2 This is a schematic diagram of the architecture of the large model based on the reuse of hybrid diversity experts provided by the present invention, as shown below. Figure 2 As shown, the code-based large model in this embodiment is a large model based on the reuse of hybrid diversity experts. The code-based large model is constructed by adding a shared expert and M ordinary experts to the dense model. The shared expert reuses the parameters of the FFN module of the dense model. The shared expert is used to extract general knowledge of lexical units. The ordinary experts are initialized with random initialization. The ordinary experts are used to extract specific knowledge of lexical units, where M is an integer greater than or equal to 2.

[0045] The multi-expert diversity initialization strategy disclosed in this invention is a strategy designed to enhance the diversity among experts, namely a diversity multi-reuse mechanism. This mechanism reuses the parameters of the FFN module of the original dense model by copying the parameters of shared experts, while randomly initializing other ordinary experts to increase diversity. This method differs from the original multi-expert large model initialization method. While reusing the capabilities of the dense model, this method avoids the problem of experts being completely identical after initialization, increases the diversity among experts, reduces redundancy between information extracted by different experts, and improves the accuracy of the generated code.

[0046] In some embodiments, the large code model further includes a routing network and a summation module;

[0047] The routing network is used to send each token to be processed to the corresponding N general experts; N is an integer greater than or equal to 2, and N is less than or equal to M;

[0048] The summation module is used to sum the lexical units output by the N ordinary experts and the lexical units output by the shared expert.

[0049] Specifically, such as Figure 2 As shown in the embodiments of this application, the code big model also includes a routing network (Router) and a summation module.

[0050] The routing network is used to send each token to be processed to the corresponding N general experts; N is an integer greater than or equal to 2, and N is less than or equal to M.

[0051] The summation module is used to sum the lexical units output by the N ordinary experts and the lexical units output by the shared expert.

[0052] In some embodiments, summing the lexical units output by the N ordinary experts and the lexical units output by the shared expert includes:

[0053] The word units output by the N ordinary experts are weighted and summed to obtain the summation result;

[0054] The summation result and the tokens output by the shared expert are summed.

[0055] Specifically, such as Figure 2 As shown in the embodiments of this application, the summation module includes a weighted summation unit and a summation unit.

[0056] The weighted summation unit performs a weighted summation on the word units output by the N ordinary experts to obtain the summation result.

[0057] The summation unit sums the summation result and the tokens output by the shared expert.

[0058] In some embodiments, the routing network is used to determine the score of each general expert based on the weight matrix of each general expert, and to determine the N general experts with the highest scores for processing word units.

[0059] Specifically, in this embodiment of the application, the routing network is used to determine the score of each ordinary expert based on the weight matrix of each ordinary expert. The calculation formula is as follows:

[0060]

[0061] in, This represents the score of a regular expert. This represents the weight matrix for ordinary experts. Indicates the word to be processed. This indicates the bias term.

[0062] The selected top-N sets of general expert indexes are as follows:

[0063]

[0064] The output of the summation module is as follows:

[0065]

[0066] in, This indicates the output of the summation module. This indicates the shared expert's output. This represents the weight value corresponding to the j-th ordinary expert. This represents the output of the j-th ordinary expert.

[0067] The code generation method based on a multi-reuse, hybrid diversity expert model provided by this invention adds a shared expert and multiple ordinary experts to the dense model to construct a large code model. By adopting a diversity multi-reuse mechanism, the differences between experts are increased, the redundancy between information extracted by different experts is reduced, and the accuracy of the generated code is improved.

[0068] In some embodiments, the method further includes:

[0069] Determine the differences in lexical output between shared experts and ordinary experts;

[0070] The diversity loss function value is determined based on the differences in lexical outputs of the shared experts and the differences in lexical outputs of the ordinary experts;

[0071] The total loss is determined based on the value of the diversity loss function.

[0072] The training of the large code model is guided by the total loss.

[0073] Specifically, in the embodiments of this application, during the training of the large code model, the training of the large code model is guided by diversity loss.

[0074] First, calculate the lexical output difference of the shared experts. The calculation formula is as follows:

[0075]

[0076] in, The difference in output after the representative lexical input is fed into the shared expert. Representative shared expert processing each word element The subsequent output, Representative shared expert processing j each word element The subsequent output, t This represents the number of tokens in the input.

[0077] Then, the difference in word output by a regular expert is calculated using the following formula:

[0078]

[0079] in, Representing the k Differences in word output among ordinary experts The differences in word output representing all ordinary experts, Representing the k A regular expert handles the first each word element The subsequent output, Representing the k A regular expert handles the first j each word element The subsequent output, The total number of experts represented n This represents the number of tokens in the input.

[0080] Then calculate the diversity loss function value. The specific calculation method is as follows:

[0081]

[0082] In this embodiment of the application, the loss function can be set as follows: Compare Larger differences between experts encourage the extraction of common knowledge, while larger differences between ordinary experts encourage the extraction of more domain or task-specific knowledge.

[0083] Then, the diversity loss is added as an additional loss to the overall loss function of the MoE large-scale model. Typically, the loss function of the MoE structural large-scale model... It consists of two parts, as shown below:

[0084]

[0085] That is, the loss function of the large model Balance loss between experts The aforementioned loss of diversity Add to the final total loss The calculation formula is as follows:

[0086]

[0087] In the actual training process, α was set to 0.01 and β was set to 0.0001.

[0088] The multi-expert diversity loss function in this embodiment is an additional loss function designed to increase the diversity among experts. This loss function guides the model to output smaller values ​​for tokens processed by shared experts than those processed by ordinary experts. This is because shared experts tend to extract general knowledge, while ordinary experts may be more inclined to extract domain-specific or task-specific knowledge. This guides experts to complete the knowledge extraction division of labor while further increasing the diversity among experts and improving the accuracy of the generated code.

[0089] The following specific example further illustrates the above method:

[0090] In this example, the DeepSeek-Coder-Base 1.3B model is used as the dense model, that is, the DeepSeek-Coder-Base 1.3B model is used as the basis of the large code model. On the basis of the DeepSeek-Coder-Base 1.3B model, one shared expert and seven ordinary experts are added to construct the large code model. N is set to 5, that is, the output results of the five ordinary experts with the highest scores are selected.

[0091] In this example, experiments were conducted on the code evaluation datasets HumanEval and HumanEval+. The accuracy comparison of the code output by different models is shown in Table 1.

[0092] Table 1. Accuracy of the code output by different models

[0093]

[0094] In the table, DeepSeek-Coder-Base 1.3B is an open-source 1B-scale code LLM, SU-MoE represents an LLM that adopts an up-reuse hybrid expert architecture, and MoDE is a large model based on up-reuse hybrid diversity experts in this application.

[0095] By comparison, the method of the present invention can achieve a relative performance improvement of more than 2.7% and 3.5% on HumanEval and HumanEval+, respectively, proving its effectiveness in code generation tasks.

[0096] In addition, the method of the present invention improves model performance without significantly increasing the burden of model training or affecting the convergence speed of model training.

[0097] In this embodiment of the application, during the training of the large code model, evol-codealpaca-v1 is selected as the training set for the code generation task. Then, the large code model is trained and fine-tuned. The main training parameter configurations are shown in Table 2.

[0098] Table 2 Training Parameter Configuration Table

[0099]

[0100] The code generation apparatus based on the multi-reuse hybrid diversity expert model provided by the present invention is described below. The code generation apparatus based on the multi-reuse hybrid diversity expert model described below and the code generation method based on the multi-reuse hybrid diversity expert model described above can be referred to in correspondence.

[0101] Figure 3 This is a schematic diagram of the structure of the code generation device based on the multi-reuse hybrid diversity expert large model provided by the present invention, as shown below. Figure 3 As shown, the code generation device based on a multi-reusable hybrid diversity expert large model provided by the present invention includes:

[0102] Module 301 is used to obtain code description information;

[0103] The generation module 302 is used to input the code description information into the trained code model and obtain the code output by the code model;

[0104] The large code model is a large model based on the reuse of hybrid diversity experts; the large code model is constructed by adding one shared expert and M ordinary experts to the dense model; the shared expert reuses the parameters of the FFN module of the dense model; the shared expert is used to extract general knowledge of lexical units; the ordinary experts are initialized with random initialization; the ordinary experts are used to extract special knowledge of lexical units, where M is an integer greater than or equal to 2.

[0105] Specifically, the code generation apparatus based on the multivariate expert model based on the above-reusable hybrid diversity expert model provided in this application embodiment can implement all the method steps implemented in the above-reusable multivariate expert model based code generation method embodiment, and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0106] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a code generation method based on a multi-reusable hybrid diversity expert large model, the method including:

[0107] Get code description information;

[0108] The code description information is input into the trained code model, and the code output by the code model is obtained.

[0109] The large code model is a large model based on the reuse of hybrid diversity experts; the large code model is constructed by adding one shared expert and M ordinary experts to the dense model; the shared expert reuses the parameters of the FFN module of the dense model; the shared expert is used to extract general knowledge of lexical units; the ordinary experts are initialized with random initialization; the ordinary experts are used to extract special knowledge of lexical units, where M is an integer greater than or equal to 2.

[0110] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0111] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the code generation method based on the multi-reuse hybrid diversity expert large model provided by the above methods, the method comprising:

[0112] Get code description information;

[0113] The code description information is input into the trained code model, and the code output by the code model is obtained.

[0114] The large code model is a large model based on the reuse of hybrid diversity experts; the large code model is constructed by adding one shared expert and M ordinary experts to the dense model; the shared expert reuses the parameters of the FFN module of the dense model; the shared expert is used to extract general knowledge of lexical units; the ordinary experts are initialized with random initialization; the ordinary experts are used to extract special knowledge of lexical units, where M is an integer greater than or equal to 2.

[0115] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the code generation method based on the multi-reusable hybrid diversity expert large model provided by the methods described above, the method comprising:

[0116] Get code description information;

[0117] The code description information is input into the trained code model, and the code output by the code model is obtained.

[0118] The large code model is a large model based on the reuse of hybrid diversity experts; the large code model is constructed by adding one shared expert and M ordinary experts to the dense model; the shared expert reuses the parameters of the FFN module of the dense model; the shared expert is used to extract general knowledge of lexical units; the ordinary experts are initialized with random initialization; the ordinary experts are used to extract special knowledge of lexical units, where M is an integer greater than or equal to 2.

[0119] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0120] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0121] It should also be noted that "determining B based on A" in the embodiments of this application means that factor A must be considered when determining B. It is not limited to "B can be determined based solely on A," but should also include: "determining B based on A and C," "determining B based on A, C, and E," "determining C based on A, and further determining B based on C," etc. It can also include using A as a condition for determining B, for example, "when A meets the first condition, use the first method to determine B"; or "when A meets the second condition, determine B," etc.; or "when A meets the third condition, determine B based on the first parameter," etc. Of course, it can also be a condition where A is a factor in determining B, for example, "when A meets the first condition, use the first method to determine C, and further determine B based on C," etc.

[0122] In the embodiments of this application, the term "multiple" refers to two or more, and other quantifiers are similar.

[0123] In the embodiments of this application, the terms "first," "second," etc., are used to distinguish similar objects, and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, and the number of objects is not limited; for example, the first object can be one or more.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A code generation method based on a multi-reusable, hybrid, diverse expert large model, characterized in that, include: Obtain code description information; the code description is a natural language description input by the user; the code description information includes at least one of functional description information, syntax description information, and personalized coding style requirements; The code description information is input into the trained code model, and the code output by the code model is obtained. The code model is a large model based on the reuse of hybrid diversity experts; The large code model is constructed by adding one shared expert and M ordinary experts to the dense model; the shared expert reuses the parameters of the FFN module of the dense model; the shared expert is used to extract general knowledge of lexical units; the ordinary experts are initialized with parameters using random initialization. The specialized knowledge used by the general experts to extract lexical units, where M is an integer greater than or equal to 2; The training of the large code model includes: Based on the output of the shared expert after processing different lexical units, the differences in the lexical unit outputs of the shared expert are determined; Based on the output of the ordinary expert after processing different lexical units, the differences in the lexical unit output of the ordinary expert are determined; The diversity loss function value is determined by using the constraint that the difference in word output of the ordinary expert is greater than the difference in word output of the shared expert; The total loss is determined based on the value of the diversity loss function. The training of the large code model is guided by the total loss. The determination of the diversity loss function value based on the lexical output differences of the shared expert and the lexical output differences of the ordinary expert includes: The difference in the shared expert's lexical output is calculated using the following formula: Among them, D s The difference in output after the representative lexical input is fed into the shared expert, f S (x i ) represents the shared expert processing the i-th word element x i The subsequent output, f S (x j ) represents the shared expert processing the j-th word element x j The subsequent output, t represents the number of words in the input; The formula for calculating the word output difference of a regular expert is as follows: Among them, D Ek D represents the difference in word output of the k-th ordinary expert. O f represents the difference in word output among all ordinary experts Ek (x i ) represents the k-th ordinary expert processing the i-th word unit x. i The subsequent output, f Ek (x j ) represents the k-th ordinary expert processing the j-th word element x. j In the subsequent output, e represents the total number of experts, and n represents the number of input tokens; Calculate the diversity loss function value L diverse The specific calculation formula is as follows: In the formula, D is set O Greater than D s ; The determination of the total loss based on the diversity loss function value includes: The diversity loss function value is added as an additional loss to the overall loss function L of the hybrid expert large model to determine the total loss L. total The calculation formula is as follows: L=L lm +αL balance L total =L lm +αL balance +βL diverse In the formula, L lm L represents the loss function of a large model. balance L represents the balance loss among experts. diverse This represents the diversity loss function value, where α and β are preset parameters.

2. The code generation method based on a multi-reusable hybrid diversity expert large model according to claim 1, characterized in that, The large code model also includes a routing network and a summation module; The routing network is used to send each token to be processed to the corresponding N general experts; N is an integer greater than or equal to 2, and N is less than or equal to M; The summation module is used to sum the lexical units output by the N ordinary experts and the lexical units output by the shared expert.

3. The code generation method based on a multi-reusable hybrid diversity expert large model according to claim 2, characterized in that, Summing the lexical units output by the N ordinary experts and the lexical units output by the shared expert includes: The word units output by the N ordinary experts are weighted and summed to obtain the summation result; The summation result and the tokens output by the shared expert are summed.

4. The code generation method based on a multi-reusable hybrid diversity expert large model according to claim 2, characterized in that, The routing network is used to determine the score of each general expert based on the weight matrix of each general expert, and to determine the N general experts with the highest scores for processing word units.

5. The code generation method based on a multi-reusable hybrid diversity expert large model according to any one of claims 1 to 4, characterized in that, The dense model is the DeepSeek-Coder-Base1.3B model.

6. A code generation device based on a multi-reusable hybrid diversity expert large model, characterized in that, include: The acquisition module is used to acquire code description information; the code description is a natural language description input by the user; the code description information includes at least one of functional description information, syntax description information, and personalized coding style requirements; A generation module is used to input the code description information into a trained code model and obtain the code output by the code model. The code model is a large model based on the reuse of hybrid diversity experts; The large code model is constructed by adding one shared expert and M ordinary experts to the dense model; the shared expert reuses the parameters of the FFN module of the dense model; the shared expert is used to extract general knowledge of lexical units; the ordinary experts are initialized with parameters using random initialization. The specialized knowledge used by the general experts to extract lexical units, where M is an integer greater than or equal to 2; The training of the large code model includes: Based on the output of the shared expert after processing different lexical units, the differences in the lexical unit outputs of the shared expert are determined; Based on the output of the ordinary expert after processing different lexical units, the differences in the lexical unit output of the ordinary expert are determined; The diversity loss function value is determined by using the constraint that the difference in word output of the ordinary expert is greater than the difference in word output of the shared expert; The total loss is determined based on the value of the diversity loss function. The training of the large code model is guided by the total loss. The determination of the total loss based on the diversity loss function value includes: First, the lexical output differences of the shared experts are calculated using the following formula: Among them, D s The difference in output after the representative lexical input is fed into the shared expert, f S (x i ) represents the shared expert processing the i-th word element x i The subsequent output, f S (x j ) represents the shared expert processing the j-th word element x j The subsequent output, t represents the number of words in the input; Then, the difference in word output by a regular expert is calculated using the following formula: Among them, D Ek D represents the difference in word output of the k-th ordinary expert. O f represents the difference in word output among all ordinary experts Ek (x i ) represents the k-th ordinary expert processing the i-th word unit x. i The subsequent output, f Ek (x j ) represents the k-th ordinary expert processing the j-th word element x. j In the subsequent output, e represents the total number of experts, and n represents the number of input tokens; Then calculate the diversity loss function value L. diverse The specific calculation formula is as follows: In the formula, D is set O Greater than D s ; The determination of the total loss based on the diversity loss function value includes: The diversity loss function value is added as an additional loss to the overall loss function L of the hybrid expert large model to determine the total loss L. total The calculation formula is as follows: L=L lm +αL balance L total =L lm +αL balance +βL diverse In the formula, L lm L represents the loss function of a large model. balance L represents the balance loss among experts. diverse This represents the diversity loss function value, where α and β are preset parameters.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the code generation method based on the multivariate expert big model as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the code generation method based on the multivariate expert big model as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the code generation method based on the multivariate expert big model as described in any one of claims 1 to 5.

Images