Code field instruction data processing method and device, storage medium, program product and computer equipment

Abstract

The invention discloses a code field instruction data processing method and device, a storage medium, a program product and computer equipment. The method comprises the steps of obtaining code instruction data; determining code instruction difficulty corresponding to the code instruction data; determining instruction following difficulty corresponding to the code instruction data; determining instruction complexity corresponding to the code instruction data based on the code instruction difficulty and the instruction following difficulty; based on the instruction complexity, screening out a target code instruction from the code instruction data; based on the target code instruction, the code instruction data set is constructed or updated, so that the instruction complexity can be evaluated in combination with the code instruction difficulty and the instruction following difficulty corresponding to the code instruction data so as to evaluate the quality of the code instruction data, and then the high-quality target code instruction can be screened out from the code instruction data. And the code instruction data set is constructed or dynamically updated, so that the quality of the code instruction data set is improved, and the optimization effect on the large model is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, storage medium, program product, and computer equipment for processing instruction data in the code domain. Background Technology

[0002] With the rapid development of artificial intelligence technology, automatic code generation has become an important research direction for improving software development efficiency. To quickly optimize the code domain capabilities of large-scale models, a large amount of code instruction data is needed to strengthen the code domain knowledge of these models, thereby enhancing their ability to generate code within the specified domain.

[0003] In related technologies, code instruction data typically includes open-source code instructions (such as code instruction data generated from large models) and user-uploaded code instructions. However, the quality of these code instructions varies greatly, and the code quality is generally low. Directly using these code instructions to optimize large models results in low optimization effectiveness.

[0004] Therefore, improving the quality of code instruction data to enhance the optimization effect on large models has become a technical problem that needs to be solved. Summary of the Invention

[0005] To address the aforementioned technical problems, embodiments of this application propose a code domain instruction data processing method, apparatus, storage medium, program product, and computer equipment, which can improve the quality of code instruction datasets to enhance the optimization effect on large models.

[0006] In a first aspect, embodiments of this application provide a code domain instruction data processing method, including: Retrieve code instruction data; Determine the code instruction difficulty corresponding to the code instruction data; Determine the instruction following difficulty corresponding to the code instruction data; Based on the difficulty of the code instruction and the difficulty of following the instruction, the instruction complexity corresponding to the code instruction data is determined; Based on the complexity of the instructions, target code instructions are selected from the code instruction data; Based on the target code instructions, construct or update the code instruction dataset.

[0007] Optionally, determining the code instruction difficulty corresponding to the code instruction data includes: Based on the code instruction data, determine its score in at least one dimension and the probability corresponding to the score, wherein the at least one dimension includes at least one of the following: a dimension related to the code logic level, and a dimension related to the code execution steps; The difficulty of the code instruction is determined based on the score and the probability.

[0008] Optionally, the code instruction data includes an instruction field, an input field, and a desired output field; Determining the instruction following difficulty corresponding to the code instruction data includes: Invoke at least one model and perform inference based on the instruction field and the input field respectively to obtain the inference result corresponding to each of the at least one model; Based on the inference results and the expected output fields, determine the first confusion level corresponding to each of the at least one model; Call the at least one model and perform continuation prediction based on the expected output field to obtain the prediction results corresponding to each of the at least one model; Based on the prediction results and the expected output field, determine the second confusion degree corresponding to each of the at least one model; The difficulty of following the instruction is determined based on the first level of confusion and the second level of confusion.

[0009] Optionally, the target code instruction is used to update the code instruction dataset, and the at least one model includes a baseline model, a first model, and a second model. The first model is obtained by optimizing the baseline model using the code instruction dataset before the update, and the second model is obtained by optimizing the first model or the baseline model using the code instruction data and the code instruction dataset before the update. The step of determining the instruction following difficulty based on the first level of confusion and the second level of confusion includes: The first cross-model instruction following degree is determined based on the ratio of the first confusion degree corresponding to the first model and the benchmark model respectively. The second cross-model instruction following degree is determined based on the ratio of the first confusion degree corresponding to the second model and the first model, respectively. The third cross-model instruction following degree is determined based on the ratio of the first confusion degree corresponding to the second model and the benchmark model respectively. The fourth cross-model instruction follow-up degree is determined based on the ratio of the first cross-model instruction follow-up degree to the second cross-model instruction follow-up degree. The first single-model instruction following degree is determined based on the ratio of the first confusion degree to the second confusion degree corresponding to the benchmark model. The instruction follow-up degree of the second single model is determined based on the ratio of the first confusion degree to the second confusion degree corresponding to the first model. The instruction follow-up degree of the third single model is determined based on the ratio of the first confusion degree to the second confusion degree corresponding to the second model. The instruction following difficulty is determined based on the first cross-model instruction following degree, the second cross-model instruction following degree, the third cross-model instruction following degree, the fourth cross-model instruction following degree, the first single-model instruction following degree, the second single-model instruction following degree, and the third single-model instruction following degree.

[0010] Optionally, the code instruction data includes multiple instructions, and the instruction complexity includes the instruction complexity of each of the multiple instructions; The step of determining the instruction complexity corresponding to the code instruction data based on the code instruction difficulty and the instruction following difficulty includes: Based on the difficulty of the code instruction and the difficulty of following the instruction, the code instruction data is clustered to obtain the clustering results; Based on the clustering results, the instruction complexity of each of the multiple instructions is determined.

[0011] Optionally, the code instruction data includes multiple instructions, the instruction complexity includes the instruction complexity of each of the multiple instructions, and the target code instruction includes a first target code instruction and a second target code instruction; The step of filtering target code instructions from the code instruction data based on the instruction complexity includes: Based on the instruction complexity of each of the plurality of instructions, the plurality of instructions are divided into at least two groups of instructions, wherein the at least two groups of instructions include at least a complex instruction group and a simple instruction group; The first target code instruction is obtained by filtering from the complex instruction group according to a first ratio; The second target code instruction is obtained by filtering from the simple instruction group according to a second ratio, wherein the second ratio is less than the first ratio.

[0012] Secondly, embodiments of this application provide a code domain instruction data processing apparatus, comprising: The data acquisition module is used to acquire code instruction data; The code instruction difficulty determination module is used to determine the code instruction difficulty corresponding to the code instruction data; The instruction following difficulty determination module is used to determine the instruction following difficulty corresponding to the code instruction data; The instruction complexity determination module is used to determine the instruction complexity corresponding to the code instruction data based on the code instruction difficulty and the instruction following difficulty. The target code instruction filtering module is used to filter target code instructions from the code instruction data based on the instruction complexity. The dataset processing module is used to construct or update the code instruction dataset based on the target code instructions.

[0013] Thirdly, embodiments of this application provide a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in any of the above-mentioned embodiments.

[0014] Fourthly, embodiments of this application provide a computer program product, including computer instructions that, when executed by a processor, implement the steps of the method described in any of the above-described embodiments.

[0015] Fifthly, embodiments of this application provide a computer device including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the steps of the method described in any of the preceding claims.

[0016] In summary, the embodiments of this application have at least the following beneficial effects: By employing the embodiments of this application, code instruction data is acquired; the code instruction difficulty corresponding to the code instruction data is determined; the instruction following difficulty corresponding to the code instruction data is determined; based on the code instruction difficulty and the instruction following difficulty, the instruction complexity corresponding to the code instruction data is determined; based on the instruction complexity, target code instructions are selected from the code instruction data; based on the target code instructions, a code instruction dataset is constructed or updated. In this way, the instruction complexity can be evaluated by combining the code instruction difficulty and instruction following difficulty corresponding to the code instruction data, thereby assessing the quality of the code instruction data. This allows for the selection of high-quality target code instructions from the code instruction data, and the construction or dynamic updating of the code instruction dataset, improving the quality of the code instruction dataset and thus enhancing the optimization effect on large models. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the code domain instruction data processing method provided in an embodiment of this application; Figure 2 This is a schematic diagram of a dialogue instruction template related to the code logic level provided in an embodiment of this application; Figure 3 This is a schematic diagram of a dialog instruction template related to code execution steps provided in an embodiment of this application; Figure 4 This is a schematic diagram of code domain instruction data processing provided in an embodiment of this application; Figure 5 This is yet another schematic diagram of code domain instruction data processing provided in the embodiments of this application; Figure 6 This is a schematic diagram of the various sets of instructions provided in the embodiments of this application; Figure 7 This is a schematic diagram illustrating the code domain model optimization provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of the code domain instruction data processing device provided in the embodiments of this application; Figure 9 This is a schematic diagram of the structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments / examples are only a part of the embodiments / examples of this application, and not all of the embodiments / examples. Based on the embodiments / examples in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0019] In the description of this application, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," "third," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "multiple" means two or more. In the description of this application, the term "comprising" and its variations are open-ended, meaning "including but not limited to." The term "based on" means "at least partially based on." The term "according to" means "at least partially according to." The term "one embodiment / example" means "at least one embodiment / example"; the term "another embodiment / example" means "at least one additional embodiment / example"; the term "some embodiments / examples" means "at least some embodiments / examples."

[0020] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0021] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this application is for the purpose of describing specific embodiments only and is not intended to limit the application. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0022] Firstly, see [the following] Figure 1 The diagram shows a flowchart of a code domain instruction data processing method provided in an embodiment of this application. This code domain instruction data processing method can be applied to a computer device with data processing capabilities. The method includes S101-S106, as detailed below.

[0023] S101, retrieve code instruction data.

[0024] In some examples, the code instruction data may be obtained by cleaning initial code instruction data. For example, this data cleaning process may include data deduplication. The initial code instruction data may include open-source code instructions (such as code instruction data generated through a large model) and / or user-uploaded code instructions. Furthermore, this data cleaning process may also include filtering the initial code instruction data for standardization and correctness based on syntax trees and / or static code analysis tools.

[0025] S102, determine the code instruction difficulty corresponding to the code instruction data.

[0026] In some examples, the code instruction difficulty can be used to indicate the complexity of each instruction included in the code instruction data. In other words, the code instruction difficulty can be obtained by analyzing the complexity of each instruction included in the code instruction data (e.g., analyzing the number of lines of code).

[0027] S103, determine the instruction following difficulty corresponding to the code instruction data.

[0028] It should be noted that instruction following refers to the ability of a pre-trained model to generate corresponding output according to the requirements indicated by the instruction after understanding the user's instruction in natural language. Therefore, the difficulty of instruction following can be used to quantify the magnitude of this ability.

[0029] S104, Based on the code instruction difficulty and the instruction following difficulty, determine the instruction complexity corresponding to the code instruction data.

[0030] In some examples, instruction complexity can be obtained by weighting the difficulty of the code instruction and the difficulty of following the instruction.

[0031] S105, Based on the instruction complexity, select the target code instruction from the code instruction data.

[0032] In some examples, the filtered target code instructions may include instructions with high complexity (e.g., above a preset complexity threshold).

[0033] S106, Based on the target code instructions, construct or update the code instruction dataset.

[0034] In some examples, a code instruction dataset can be constructed based on the target code instructions, or an already constructed code instruction dataset can be iteratively updated based on the target code instructions.

[0035] In one optional implementation, determining the code instruction difficulty corresponding to the code instruction data includes: Based on the code instruction data, determine its score in at least one dimension and the probability corresponding to the score, wherein the at least one dimension includes at least one of the following: a dimension related to the code logic level, and a dimension related to the code execution steps; The difficulty of the code instruction is determined based on the score and the probability.

[0036] In some examples, code instruction data may include an instruction field (such as "Please explain the following code"), an input field (Input), and an expected output field (Output); the input field can be information entered by the user, such as the code that the user enters to be scored, and the output field can be the expected output result.

[0037] In some examples, model input instructions can be constructed based on the Instruct and Input fields in the code instruction data. For instance, the model input instructions can be obtained by directly concatenating the Instruct and Input fields. Alternatively, see [link to relevant documentation]. Figure 2 , Figure 3 , as Figure 2 , Figure 3 The dialogue instruction template shown is used to concatenate the instruction field (Instruct) and the input field (Input); and the input instruction to the model is scored in at least one of the above dimensions. For example, it can be scored based on... Figure 2 , Figure 3 The input command constructed using the dialogue command template shown is input into a preset large language model to score the input command in at least one of the aforementioned dimensions, and to obtain the probability corresponding to the score. This probability can be used to indicate the confidence level of the corresponding score. Figure 2The template shown can be used to represent scores related to the code's logical hierarchy. Figure 3 The template shown can be used to represent scores related to code execution steps.

[0038] In some examples, dimensions related to the code's logical hierarchy can be used to indicate the number of logical branches contained in an instruction, such as conditional statements, loops, and other branches that can increase the complexity of the instruction.

[0039] In some examples, dimensions related to code execution steps can be used to indicate the number of steps required to implement an instruction; the more steps, the more complex the instruction is typically.

[0040] In some examples, suppose the scores for n dimensions are denoted as W1, W2, ..., Wn, and the probabilities corresponding to the scores for each of the n dimensions are P1, P2, ..., Pn. Taking the first dimension as an example, if the probability corresponding to this dimension is less than 0.5, we can take the average score of this dimension as S1 = 2.5. If the probability corresponding to this dimension is greater than 0.5, we can take S1 = W1 * P1. The final score S is represented as follows: S = S1 + S2 + ... + Sn. Thus, the final score S can be used to represent the difficulty of the code instruction.

[0041] In some examples, the score for each dimension can be obtained by independently evaluating each dimension in one or more steps based on multiple evaluation models.

[0042] In one optional implementation, the code instruction data includes an instruction field, an input field, and a desired output field; Determining the instruction following difficulty corresponding to the code instruction data includes: Invoke at least one model and perform inference based on the instruction field and the input field respectively to obtain the inference result corresponding to each of the at least one model; Based on the inference results and the expected output fields, determine the first confusion level corresponding to each of the at least one model; Call the at least one model and perform continuation prediction based on the expected output field to obtain the prediction results corresponding to each of the at least one model; Based on the prediction results and the expected output field, determine the second confusion degree corresponding to each of the at least one model; The difficulty of following the instruction is determined based on the first level of confusion and the second level of confusion.

[0043] In some examples, the model input instruction (i.e., Instruct + Input) can be constructed based on the instruction field and the input field, and this model input instruction can be input into each of the at least one of the models to obtain the inference result output by each model and its corresponding inference result. This inference result can be used to characterize the prediction probability matrix.

[0044] In some examples, the preceding content in the expected output field can be input into each of the at least one of the above models for continuation prediction, resulting in the corresponding prediction result output by each model. This prediction result can be used to characterize the prediction probability matrix. In this embodiment, it is equivalent to removing the model input instructions (i.e., Instruct + Input) and using only the expected output field (Output) for continuation and recalculating the corresponding confusion.

[0045] In some examples, perplexity can be calculated by using a prediction probability matrix and the expected output field, obtaining the probability value of the expected output field on the corresponding prediction probability matrix, and then calculating the perplexity. This perplexity represents the difficulty of generating the corresponding expected output field if the current instruction is to execute. The formula for calculating perplexity is as follows;

[0046] in, This can be the length of the Output (e.g., how many tokens it consists of). It can be a model predicting prior knowledge. Content (i.e.) When generating the first Content (i.e.) The probability of ) It can be used to indicate the probability value of the expected output field on the corresponding prediction probability matrix.

[0047] In one optional implementation, the target code instructions are used to update the code instruction dataset, and the at least one model includes a baseline model, a first model, and a second model. The first model is obtained by optimizing the baseline model using the code instruction dataset before the update, and the second model is obtained by optimizing the first model or the baseline model using the code instruction data and the code instruction dataset before the update. The step of determining the instruction following difficulty based on the first level of confusion and the second level of confusion includes: The first cross-model instruction following degree is determined based on the ratio of the first confusion degree corresponding to the first model and the benchmark model respectively. The second cross-model instruction following degree is determined based on the ratio of the first confusion degree corresponding to the second model and the first model, respectively. The third cross-model instruction following degree is determined based on the ratio of the first confusion degree corresponding to the second model and the benchmark model respectively. The fourth cross-model instruction follow-up degree is determined based on the ratio of the first cross-model instruction follow-up degree to the second cross-model instruction follow-up degree. The first single-model instruction following degree is determined based on the ratio of the first confusion degree to the second confusion degree corresponding to the benchmark model. The instruction follow-up degree of the second single model is determined based on the ratio of the first confusion degree to the second confusion degree corresponding to the first model. The instruction follow-up degree of the third single model is determined based on the ratio of the first confusion degree to the second confusion degree corresponding to the second model. The instruction following difficulty is determined based on the first cross-model instruction following degree, the second cross-model instruction following degree, the third cross-model instruction following degree, the fourth cross-model instruction following degree, the first single-model instruction following degree, the second single-model instruction following degree, and the third single-model instruction following degree.

[0048] In some examples, see Figure 4 Assuming the baseline model is M0, the first model is M1, the second model is M1-1, the code instruction dataset before the update is the SFT instruction dataset, and the code instruction data is the newly added evaluation sample, then the baseline model, the first model, and the second model can be evaluated using PPL to obtain their respective confusion levels (first and second confusion levels).

[0049] In some examples, see Figure 5 Assuming the first confusion factor for the baseline model is PPL-00 and the second confusion factor is PPL-01, the first confusion factor for the first model is PPL-10 and the second confusion factor is PPL-11, and the first confusion factor for the second model is PPL-20 and the second confusion factor is PPL-21, then: The first cross-model instruction follow-up degree Delta1 can be expressed as: Delta1 = PPL-10 / PPL-00; The second cross-model instruction following degree Delta2 can be expressed as: Delta2 = PPL-20 / PPL-10; The third cross-model instruction follow-up degree Delta3 can be expressed as: Delta3 = PPL-20 / PPL-00; The fourth cross-model instruction following degree Delta4 can be expressed as: Delta4 = Delta1 / Delta2.

[0050] Delta1-3 represent the degree of change in PPL after training at different stages. The greater the decrease in PPL, the simpler the instruction is to be followed during computation for the corresponding model. Delta4 represents the change in the impact on instructions after training with a small amount of data. The more drastic the change, the simpler the instruction is.

[0051] Furthermore, the change in PPL after removing model input instructions can be evaluated to verify the difficulty of the model instructions themselves and the difficulty of following the related code data. For example: The first single-model instruction follow-up degree Delta1-1 can be expressed as: Delta1-1=PPL-00 / PPL-01; The second single-model instruction following degree Delta2-1 can be expressed as: Delta2-1=PPL-10 / PPL-11; The third single-model instruction following degree Delta3-1 can be expressed as: Delta3-1=PPL-20 / PPL-21.

[0052] At least some of the above Delta values ​​can be added together to get the final instruction follow difficulty.

[0053] In one optional implementation, the code instruction data includes multiple instructions, and the instruction complexity includes the instruction complexity of each of the multiple instructions; The step of determining the instruction complexity corresponding to the code instruction data based on the code instruction difficulty and the instruction following difficulty includes: Based on the difficulty of the code instruction and the difficulty of following the instruction, the code instruction data is clustered to obtain the clustering results; Based on the clustering results, the instruction complexity of each of the multiple instructions is determined.

[0054] In some examples, clustering can be performed based on the difficulty of code instructions and the difficulty of following instructions to obtain clustering results. These clustering results can include cluster centers and the distribution dimensions of scatter points. Thus, the instruction quality type of the corresponding cluster can be determined based on the cluster centers and the distribution dimensions of scatter points to characterize the instruction complexity. The quadrant can be determined based on the median in the corresponding cluster, or it can be calculated using empirical values ​​or other methods.

[0055] In one optional implementation, the code instruction data includes multiple instructions, the instruction complexity includes the instruction complexity of each of the multiple instructions, and the target code instruction includes a first target code instruction and a second target code instruction; The step of filtering target code instructions from the code instruction data based on the instruction complexity includes: Based on the instruction complexity of each of the plurality of instructions, the plurality of instructions are divided into at least two groups of instructions, wherein the at least two groups of instructions include at least a complex instruction group and a simple instruction group; The first target code instruction is obtained by filtering from the complex instruction group according to a first ratio; The second target code instruction is obtained by filtering from the simple instruction group according to a second ratio, wherein the second ratio is less than the first ratio.

[0056] In some examples, you can refer to Figure 6 The at least two sets of instructions may include suspected problematic instructions, fitting instructions, complex instruction groups and simple instruction groups. For suspected problematic instructions, their corresponding PPL can be calculated, and based on the difficulty of the code instruction and the difficulty of following the instruction, the suspected problematic instructions can be selected for screening based on the manual evaluation results combined with PPL or can be directly discarded, thereby adding the screened code instructions to the code instruction dataset.

[0057] In some examples, the first proportion can be 70% to 90%, such as 80%.

[0058] In some examples, the second percentage can be 10% to 30%, such as 20%.

[0059] In some examples, see Figure 7 The updated code instruction dataset can be a new SFT instruction dataset. Thus, for the second model, the code instruction data for the next round (the new evaluation samples for the next round) can be obtained and the above-mentioned related embodiments can be repeated so that the second model can be dynamically optimized into the third model.

[0060] Secondly, correspondingly, the embodiments of this application also provide a code domain instruction data processing apparatus, which can implement all the processes of the code domain instruction data processing method provided in the above embodiments.

[0061] See Figure 8 This illustration shows a schematic diagram of the structure of a code domain instruction data processing apparatus 800 provided in an embodiment of this application. The code domain instruction data processing apparatus 800 includes: Data acquisition module 801 is used to acquire code instruction data; The code instruction difficulty determination module 802 is used to determine the code instruction difficulty corresponding to the code instruction data. Instruction following difficulty determination module 803 is used to determine the instruction following difficulty corresponding to the code instruction data; The instruction complexity determination module 804 is used to determine the instruction complexity corresponding to the code instruction data based on the code instruction difficulty and the instruction following difficulty. The target code instruction filtering module 805 is used to filter target code instructions from the code instruction data based on the instruction complexity. The dataset processing module 806 is used to construct or update the code instruction dataset based on the target code instructions.

[0062] In one optional implementation, determining the code instruction difficulty corresponding to the code instruction data includes: Based on the code instruction data, determine its score in at least one dimension and the probability corresponding to the score, wherein the at least one dimension includes at least one of the following: a dimension related to the code logic level, and a dimension related to the code execution steps; The difficulty of the code instruction is determined based on the score and the probability.

[0063] In one optional implementation, the code instruction data includes an instruction field, an input field, and a desired output field; Determining the instruction following difficulty corresponding to the code instruction data includes: Invoke at least one model and perform inference based on the instruction field and the input field respectively to obtain the inference result corresponding to each of the at least one model; Based on the inference results and the expected output fields, determine the first confusion level corresponding to each of the at least one model; Call the at least one model and perform continuation prediction based on the expected output field to obtain the prediction results corresponding to each of the at least one model; Based on the prediction results and the expected output field, determine the second confusion degree corresponding to each of the at least one model; The difficulty of following the instruction is determined based on the first level of confusion and the second level of confusion.

[0064] In one optional implementation, the target code instructions are used to update the code instruction dataset, and the at least one model includes a baseline model, a first model, and a second model. The first model is obtained by optimizing the baseline model using the code instruction dataset before the update, and the second model is obtained by optimizing the first model or the baseline model using the code instruction data and the code instruction dataset before the update. The step of determining the instruction following difficulty based on the first level of confusion and the second level of confusion includes: The first cross-model instruction following degree is determined based on the ratio of the first confusion degree corresponding to the first model and the benchmark model respectively. The second cross-model instruction following degree is determined based on the ratio of the first confusion degree corresponding to the second model and the first model, respectively. The third cross-model instruction following degree is determined based on the ratio of the first confusion degree corresponding to the second model and the benchmark model respectively. The fourth cross-model instruction follow-up degree is determined based on the ratio of the first cross-model instruction follow-up degree to the second cross-model instruction follow-up degree. The first single-model instruction following degree is determined based on the ratio of the first confusion degree to the second confusion degree corresponding to the benchmark model. The instruction follow-up degree of the second single model is determined based on the ratio of the first confusion degree to the second confusion degree corresponding to the first model. The instruction follow-up degree of the third single model is determined based on the ratio of the first confusion degree to the second confusion degree corresponding to the second model. The instruction following difficulty is determined based on the first cross-model instruction following degree, the second cross-model instruction following degree, the third cross-model instruction following degree, the fourth cross-model instruction following degree, the first single-model instruction following degree, the second single-model instruction following degree, and the third single-model instruction following degree.

[0065] In one optional implementation, the code instruction data includes multiple instructions, and the instruction complexity includes the instruction complexity of each of the multiple instructions; The step of determining the instruction complexity corresponding to the code instruction data based on the code instruction difficulty and the instruction following difficulty includes: Based on the difficulty of the code instruction and the difficulty of following the instruction, the code instruction data is clustered to obtain the clustering results; Based on the clustering results, the instruction complexity of each of the multiple instructions is determined.

[0066] In one optional implementation, the code instruction data includes multiple instructions, the instruction complexity includes the instruction complexity of each of the multiple instructions, and the target code instruction includes a first target code instruction and a second target code instruction; The step of filtering target code instructions from the code instruction data based on the instruction complexity includes: Based on the instruction complexity of each of the plurality of instructions, the plurality of instructions are divided into at least two groups of instructions, wherein the at least two groups of instructions include at least a complex instruction group and a simple instruction group; The first target code instruction is obtained by filtering from the complex instruction group according to a first ratio; The second target code instruction is obtained by filtering from the simple instruction group according to a second ratio, wherein the second ratio is less than the first ratio.

[0067] Thirdly, embodiments of this application provide a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in any of the above-mentioned embodiments.

[0068] Fourthly, embodiments of this application provide a computer program product, including computer instructions that, when executed by a processor, implement the steps of the method described in any of the above-described embodiments.

[0069] Fifthly, embodiments of this application provide a computer device including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the steps of the method described in any of the preceding claims.

[0070] See Figure 9 The computer device of this embodiment includes: a processor 901, a memory 902, and a computer program stored in the memory 902 and executable on the processor 901, such as a code domain instruction data processing program. When the processor 901 executes the computer program, it implements the steps in the various code domain instruction data processing method embodiments described above, for example... Figure 1 The steps S101-S106 are shown.

[0071] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 902 and executed by the processor 901 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the computer device.

[0072] The computer device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device may include, but is not limited to, a processor 901 and a memory 902. Those skilled in the art will understand that the schematic diagram is merely an example of a computer device and does not constitute a limitation on the computer device. It may include more or fewer components than shown, or combine certain components, or different components. For example, the computer device may also include input / output devices, network access devices, buses, etc.

[0073] The processor 901 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor 901 can be any conventional processor. The processor 901 is the control center of the computer device, connecting various parts of the entire computer device through various interfaces and lines.

[0074] The memory 902 can be used to store the computer programs and / or modules. The processor 901 implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 902 and calling the data stored in the memory 902. The memory 902 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 902 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0075] Wherein, if the modules / units integrated into the computer device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a non-transitory computer-readable storage medium. When the computer program is executed by the processor 901, it can implement the steps of the various method embodiments described above. Wherein, the computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0076] In summary, the embodiments of this application have at least the following beneficial effects: By employing the embodiments of this application, code instruction data is acquired; the code instruction difficulty corresponding to the code instruction data is determined; the instruction following difficulty corresponding to the code instruction data is determined; based on the code instruction difficulty and the instruction following difficulty, the instruction complexity corresponding to the code instruction data is determined; based on the instruction complexity, target code instructions are selected from the code instruction data; based on the target code instructions, a code instruction dataset is constructed or updated. In this way, the instruction complexity can be evaluated by combining the code instruction difficulty and instruction following difficulty corresponding to the code instruction data, thereby assessing the quality of the code instruction data. This allows for the selection of high-quality target code instructions from the code instruction data, and the construction or dynamic updating of the code instruction dataset, improving the quality of the code instruction dataset and thus enhancing the optimization effect on large models.

[0077] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary hardware platforms, or it can be implemented entirely by hardware. Based on this understanding, all or part of the technical solutions of this application that contribute to the background technology can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM (Read-Only Memory) / RAM (Random Access Memory), magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0078] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A method for processing instruction data in a code domain, characterized in that, include: Retrieve code instruction data; Determine the code instruction difficulty corresponding to the code instruction data; Determine the instruction following difficulty corresponding to the code instruction data; Based on the difficulty of the code instruction and the difficulty of following the instruction, the instruction complexity corresponding to the code instruction data is determined; Based on the complexity of the instructions, target code instructions are selected from the code instruction data; Based on the target code instructions, construct or update the code instruction dataset.

2. The method according to claim 1, characterized in that, Determining the code instruction difficulty corresponding to the code instruction data includes: Based on the code instruction data, determine its score in at least one dimension and the probability corresponding to the score, wherein the at least one dimension includes at least one of the following: a dimension related to the code logic level, and a dimension related to the code execution steps; The difficulty of the code instruction is determined based on the score and the probability.

3. The method according to claim 1, characterized in that, The code instruction data includes instruction fields, input fields, and expected output fields; Determining the instruction following difficulty corresponding to the code instruction data includes: Invoke at least one model and perform inference based on the instruction field and the input field respectively to obtain the inference result corresponding to each of the at least one model; Based on the inference results and the expected output fields, determine the first confusion level corresponding to each of the at least one model; Call the at least one model and perform continuation prediction based on the expected output field to obtain the prediction results corresponding to each of the at least one model; Based on the prediction results and the expected output field, determine the second confusion degree corresponding to each of the at least one model; The difficulty of following the instruction is determined based on the first level of confusion and the second level of confusion.

4. The method according to claim 3, characterized in that, The target code instruction is used to update the code instruction dataset. The at least one model includes a baseline model, a first model, and a second model. The first model is obtained by optimizing the baseline model using the code instruction dataset before the update. The second model is obtained by optimizing the first model or the baseline model using the code instruction data and the code instruction dataset before the update. The step of determining the instruction following difficulty based on the first level of confusion and the second level of confusion includes: The first cross-model instruction following degree is determined based on the ratio of the first confusion degree corresponding to the first model and the benchmark model respectively. The second cross-model instruction following degree is determined based on the ratio of the first confusion degree corresponding to the second model and the first model, respectively. The third cross-model instruction following degree is determined based on the ratio of the first confusion degree corresponding to the second model and the benchmark model respectively. The fourth cross-model instruction follow-up degree is determined based on the ratio of the first cross-model instruction follow-up degree to the second cross-model instruction follow-up degree. The first single-model instruction following degree is determined based on the ratio of the first confusion degree to the second confusion degree corresponding to the benchmark model. The instruction follow-up degree of the second single model is determined based on the ratio of the first confusion degree to the second confusion degree corresponding to the first model. The instruction follow-up degree of the third single model is determined based on the ratio of the first confusion degree to the second confusion degree corresponding to the second model. The instruction following difficulty is determined based on the first cross-model instruction following degree, the second cross-model instruction following degree, the third cross-model instruction following degree, the fourth cross-model instruction following degree, the first single-model instruction following degree, the second single-model instruction following degree, and the third single-model instruction following degree.

5. The method according to claim 1, characterized in that, The code instruction data includes multiple instructions, and the instruction complexity includes the individual instruction complexity of each of the multiple instructions; The step of determining the instruction complexity corresponding to the code instruction data based on the code instruction difficulty and the instruction following difficulty includes: Based on the difficulty of the code instruction and the difficulty of following the instruction, the code instruction data is clustered to obtain the clustering results; Based on the clustering results, the instruction complexity of each of the multiple instructions is determined.

6. The method according to claim 1, characterized in that, The code instruction data includes multiple instructions, the instruction complexity includes the instruction complexity of each of the multiple instructions, and the target code instruction includes a first target code instruction and a second target code instruction; The step of filtering target code instructions from the code instruction data based on the instruction complexity includes: Based on the instruction complexity of each of the plurality of instructions, the plurality of instructions are divided into at least two groups of instructions, wherein the at least two groups of instructions include at least a complex instruction group and a simple instruction group; The first target code instruction is obtained by filtering from the complex instruction group according to a first ratio; The second target code instruction is obtained by filtering from the simple instruction group according to a second ratio, wherein the second ratio is less than the first ratio.

7. A code domain instruction data processing device, characterized in that, include: The data acquisition module is used to acquire code instruction data; The code instruction difficulty determination module is used to determine the code instruction difficulty corresponding to the code instruction data; The instruction following difficulty determination module is used to determine the instruction following difficulty corresponding to the code instruction data; The instruction complexity determination module is used to determine the instruction complexity corresponding to the code instruction data based on the code instruction difficulty and the instruction following difficulty. The target code instruction filtering module is used to filter target code instructions from the code instruction data based on the instruction complexity. The dataset processing module is used to construct or update the code instruction dataset based on the target code instructions.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-6.

9. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the method described in any one of claims 1-6.

10. A computer device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the method of any one of claims 1-6.

Images