Quality analysis and defect repair method for code data set

Abstract

The invention provides a quality analysis and defect repair method for a code data set, and relates to the technical field of data processing. The method comprises the following steps: acquiring a code data set for model training or adjustment; determining a characterization quality index and a content quality index of the code data set; determining a quality defect of the code data set according to the characterization quality index and the content quality index; automatic or semi-automatic repair treatment is carried out on quality defects; and generating display information about the quality defect and the comparison information before and after the quality defect is repaired. According to the method, the characterization quality index and the content quality index of the code data set for large language model training or adjustment are determined, so that the quality of the code data set is evaluated from multiple dimensions, and standardization, objectivity and comprehensiveness of code data set quality evaluation are guaranteed; by performing automatic or semi-automatic repair processing on the quality defect, the repair efficiency of the quality defect of the code data set is effectively improved, and the labor cost and the like are reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method for quality analysis and defect repair of a code data set. Background Art

[0002] With the rapid evolution of Large Language Model (LLM) technology, its demand for data is also continuously upgrading. As a collection of corresponding code samples, code datasets can provide code-related capabilities for large language models. For example, since code data is structured and logically rigorous, it can enhance the logical thinking and analysis capabilities of large language models. Therefore, controlling and improving the quality of code datasets is crucial to improving the data processing capabilities of large language models.

[0003] However, there are some problems with the current quality control techniques for code datasets, such as:

[0004] 1. There is a lack of standardized methods for evaluating the quality of code datasets, which makes it difficult for users to systematically judge the quality of a code dataset. The quality can only be determined through manual inspection. However, due to differences in individual knowledge and subjective cognition of inspectors, the evaluation is not objective enough, and manual inspection usually brings high costs.

[0005] 2. Defects in code datasets usually need to be manually inspected and repaired using professional knowledge, which requires the inspectors to have certain coding capabilities and be able to correctly identify and modify defects in samples, which is inefficient and extremely costly. Summary of the invention

[0006] In order to solve at least one aspect of the above technical problems, the present invention proposes a quality analysis and defect repair method for a code data set, comprising:

[0007] Get the code dataset for model training or tuning;

[0008] Determine a representation quality index and a content quality index of the code data set, wherein the representation quality index is used to refer to the quality of the data presentation form of the code data set, and the content quality index is used to refer to the quality of the data presentation content of the code data set;

[0009] Determining quality defects of the code data set according to the representation quality indicator and the content quality indicator;

[0010] Performing automated or semi-automated repair processing on the quality defects;

[0011] Display information about the quality defect and comparison information before and after repair of the quality defect is generated.

[0012] Optionally, the characterization quality index includes at least one of a repetition rate, a format standard rate, a sample defect rate, a code syntax correctness rate, and a code block standard rate.

[0013] Optionally, the automatic or semi-automatic repair process for the quality defect includes:

[0014] When the quality defect with respect to the repetition rate exists, deleting repeated samples in the code data set;

[0015] When the quality defect with respect to the format standard rate exists, a regular expression is used to perform repair processing on the corresponding defective sample;

[0016] When the quality defect regarding the sample defect rate exists, deleting the defective samples in the code data set;

[0017] When the quality defect regarding the code syntax correctness exists, a second preset model is used to repair the corresponding defective sample;

[0018] When the quality defect with respect to the standard rate of the code block exists, a third preset model is used to perform repair processing on the corresponding defective sample.

[0019] Optionally, the content quality indicator includes at least one of the scores of the code dataset regarding intent completeness, response correctness, code logic correctness, and question-answer correspondence.

[0020] Optionally, the determining the representation quality index and the content quality index of the code data set includes:

[0021] Using a first preset expert model to adjust the code data set with respect to the intent completeness, to obtain a first code adjustment data set; inputting the code data set and the first code adjustment data set into a preset universal scoring model, and outputting a first score of the code data set with respect to the intent completeness; determining the score of the code data set with respect to the intent completeness according to the first score obtained at least once;

[0022] and / or, using a second preset expert model to adjust the code dataset with respect to the correctness of the response to obtain a second code adjustment dataset; inputting the code dataset and the second code adjustment dataset into the preset universal scoring model, and outputting a second score of the code dataset with respect to the correctness of the response; determining the score of the code dataset with respect to the correctness of the response based on the second score obtained at least once;

[0023] and / or, using a third preset expert model to adjust the code data set with respect to the correctness of the code logic to obtain a third code adjustment data set; inputting the code data set and the third code adjustment data set into the preset universal scoring model, and outputting a third score of the code data set with respect to the correctness of the code logic; determining the score of the code data set with respect to the correctness of the code logic according to the third score obtained at least once;

[0024] And / or, using a fourth preset expert model to adjust the code dataset with respect to the question-answer correspondence to obtain a fourth code adjusted dataset; inputting the code dataset and the fourth code adjusted dataset into the preset universal scoring model, and outputting a fourth score of the code dataset with respect to the question-answer correspondence; determining the score of the code dataset with respect to the question-answer correspondence based on the fourth score obtained at least once.

[0025] Optionally, determining the score of the code dataset regarding the intent completeness according to the first score obtained at least once comprises:

[0026] Determining a score of the code dataset regarding the intent completeness based on a first variance and a first average of all the first scores;

[0027] And / or, determining the score of the code dataset regarding the correctness of the response according to the second score obtained at least once comprises:

[0028] determining a score of the code data set regarding the correctness of the response based on a second variance and a second mean of all the second scores;

[0029] And / or, determining the score of the code data set regarding the logical correctness of the code according to the third score obtained at least once comprises:

[0030] Determining a score of the code data set regarding logical correctness of the code according to a third difference and a third average of all the third scores;

[0031] And / or, determining the score of the code dataset regarding the question-answer correspondence based on the fourth score obtained at least once includes:

[0032] The score of the code dataset with respect to the question-answer correspondence is determined based on the fourth variance and the fourth average of all the fourth scores.

[0033] Optionally, determining the score of the code dataset regarding the intent completeness according to the first variance and the first average of all the first scores comprises:

[0034] When the first variance is less than or equal to a first preset threshold, the first average value is used as the score of the code data set with respect to the intent completeness; when the first variance is greater than a first preset threshold, a prompt message for performing a manual score with respect to the intent completeness is generated, and in response to receiving a first manual score with respect to the intent completeness, the first manual score is used as the score of the code data set with respect to the intent completeness;

[0035] And / or, determining the score of the code dataset regarding the correctness of the response according to the second variance and the second average of all the second scores comprises:

[0036] When the second variance is less than or equal to a second preset threshold, the second average value is used as the score of the code data set regarding the correctness of the response; when the second variance is greater than the second preset threshold, a prompt message for manual scoring of the correctness of the response is generated, and in response to receiving a second manual score regarding the correctness of the response, the second manual score is used as the score of the code data set regarding the correctness of the response;

[0037] And / or, determining the score of the code data set regarding the logical correctness of the code according to the third party differences and the third average of all the third scores includes:

[0038] When the third difference is less than or equal to a third preset threshold, the third average value is used as the score of the code data set regarding the correctness of the code logic; when the third difference is greater than the third preset threshold, a prompt message for manual scoring of the correctness of the code logic is generated, and in response to receiving a third manual score regarding the correctness of the code logic, the third manual score is used as the score of the code data set regarding the correctness of the code logic;

[0039] And / or, determining the score of the code dataset regarding the question-answer correspondence according to the fourth variance and the fourth mean of all the fourth scores includes:

[0040] When the fourth variance is less than or equal to a fourth preset threshold, the fourth average value is used as the score of the code dataset regarding the correctness of the response; when the fourth variance is greater than the fourth preset threshold, a prompt message is generated regarding performing a manual score on the question-answer correspondence, and in response to receiving a fourth manual score on the question-answer correspondence, the fourth manual score is used as the score of the code dataset regarding the question-answer correspondence.

[0041] Optionally, the automatic or semi-automatic repair process for the quality defect includes:

[0042] Step 410: When the code data set has a content quality defect regarding at least one of the intent completeness, the response correctness, the code logic correctness, and the question-answer correspondence, determine a first defect description and repair suggestion information regarding the content quality defect through a fifth preset model;

[0043] Step 420: According to the first defect description and repair suggestion information, a sixth preset model is used to repair the code data set to obtain a corrected code data set;

[0044] Step 430: Use the first preset model to score the correction code data set to obtain a second content quality score corresponding to the correction code data set, wherein the first preset model includes a first preset expert model, a second preset expert model, a third preset expert model, a fourth preset expert model and a preset general scoring model;

[0045] Step 440: If the second content quality score is higher than the first content quality score corresponding to the code data set and higher than the fifth preset threshold, determine that the corresponding content quality defect is repaired; if the second content quality score is lower than at least one of the first content quality score and the fifth preset threshold, use the corrected code data set as the code data set in step 410, and repeat steps 410 to 440 until the second content quality score corresponding to the corrected code data set is higher than the first content quality score and higher than the fifth preset threshold; wherein the first content quality score includes at least one of the scores of the code data set regarding intent completeness, response correctness, code logic correctness, and question-answer correspondence.

[0046] Optionally, after acquiring the code data set for model training or adjustment and before automatically or semi-automatically repairing the quality defects, the code data set quality analysis and defect repair method further includes:

[0047] Using a preset statistical algorithm to obtain statistical features of the code data set;

[0048] The automatic or semi-automatic repair process for the quality defect includes:

[0049] Display information about the statistical feature is generated.

[0050] To solve the above problems, the present invention further provides a quality analysis and defect repair device for a code data set, comprising:

[0051] An acquisition unit, used to acquire a code dataset for model training or adjustment;

[0052] a quality indicator determination unit, configured to determine a characterization quality indicator and a content quality indicator of the code data set, wherein the characterization quality indicator is used to refer to the quality of the data presentation form of the code data set, and the content quality indicator is used to refer to the quality of the data presentation content of the code data set;

[0053] A quality defect determination unit, configured to determine the quality defect of the code data set according to the characterization quality indicator and the content quality indicator;

[0054] A quality defect repair unit, used for automatically or semi-automatically repairing the quality defect;

[0055] A display unit is used to generate display information about the quality defect and comparison information before and after the quality defect is repaired.

[0056] To solve the above problem, the present invention further provides an electronic device, including a memory and a processor;

[0057] The memory is used to store computer programs;

[0058] The processor is used to implement the quality analysis and defect repair method of the code data set as described above when executing the computer program.

[0059] To solve the above problems, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is read and executed by a processor, the quality analysis and defect repair method of the code data set as described above is implemented.

[0060] Compared with the prior art, the present invention has the following beneficial effects: the present invention can realize the quality analysis and defect repair of the code data set used for training or adjusting a large language model, and can be used to provide a high-quality code data set for training or adjusting a large language model. Specifically, by determining the characterization quality index and content quality index of the code data set used for training or adjusting a large language model, the quality of the code data set is evaluated from multiple dimensions (or angles), the standardization (systematicity), objectivity and comprehensiveness of the quality evaluation of the code data set are guaranteed, and the applicability of the present invention is improved. According to the characterization quality index and the content quality index, the defects of the code data set are determined so as to accurately identify the quality defects, thereby facilitating targeted repairs; by performing automated or semi-automated repair processing on the quality defects, the repair efficiency of the quality defects of the code data set is effectively improved, the labor cost is reduced, and the consistency and accuracy of the quality defect repair are guaranteed. By generating display information about the quality defects and the comparison information before and after the quality defect repair, so that the user can clearly and intuitively understand the specific information of the corresponding quality defects, the effect of the quality defect repair and the improvement of the quality of the code data set, it is helpful to improve the user's confidence and understanding of the repair operation and the repaired code data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 Schematic diagram of the process of the quality analysis and defect repair method of the code data set in the embodiment of the present invention;

[0062] Figure 2 4 is a schematic diagram of a sub-flow chart of step 400 in an embodiment of the present invention;

[0063] Figure 3 It is a structural block diagram of a device for quality analysis and defect repair of a code data set according to an embodiment of the present invention;

[0064] Figure 4 FIG. 4 is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0066] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.

[0067] Combination Figure 1As shown, an embodiment of the present invention provides a method for quality analysis and defect repair of a code data set, comprising the following steps:

[0068] Step 100: Obtain a code dataset for model training or adjustment.

[0069] Specifically, the method first obtains (collects) a code dataset for training or adjusting a model such as a large language model (LLM) through step 100, such as by crawling the web, downloading open source, manually generating, or self-generating the model. The training of a large language model refers to using a large-scale text dataset to allow the model to learn the general characteristics of the language through pre-training to improve its language understanding and generation capabilities; the adjustment (or fine-tuning) of a large language model refers to further training the model in a supervised manner using a dataset related to a specific field or task on the basis of pre-training to adapt to the language and task requirements of a specific field.

[0070] Exemplarily, code datasets can be collected from various available sources (such as open source projects, online code repositories, etc.) for training or adjusting the large language model to ensure that the collected code datasets can cover different fields, different topics, and different programming languages, so as to provide the large language model with more comprehensive and rich grammatical and semantic information, thereby improving the code understanding and generation capabilities of the large language model obtained through training or adjustment of the code dataset, and improving the versatility and applicability of the large language model obtained through training or adjustment of the code dataset.

[0071] Step 200: Determine a representation quality index and a content quality index of the code data set, wherein the representation quality index is used to refer to the quality of the data presentation form of the code data set, and the content quality index is used to refer to the quality of the data presentation content of the code data set.

[0072] In order to achieve standardized evaluation of the quality of the code dataset, the method sets a representation quality index for the quality of the data presentation form of the code dataset and a content quality index for the quality of the data presentation content of the code dataset, so as to evaluate the quality of the code dataset from multiple dimensions (or angles), ensure the standardization (systematic), objectivity and comprehensiveness of the quality evaluation of the code dataset, and improve the applicability of the method. Specifically, in step 200, the code dataset obtained in step 100 is analyzed and evaluated (assessed) from the aspects of the representation quality index and the content quality index, and at the same time, a quantitative basis is provided for the defect analysis and repair of the subsequent steps of the method. Exemplarily, the representation quality index includes repetition rate, format standard rate, sample defect rate, code syntax correctness rate and code block standard rate, etc.; the content quality index includes intention completeness, response correctness, code logic correctness and question-answer correspondence, etc.

[0073] Step 300: Determine the quality defects of the code data set according to the characterization quality index and the content quality index.

[0074] Specifically, according to the characterization quality index and content quality index determined in step 200, the part that does not meet the specification is determined (or identified) as a quality defect of the code data set (recorded as a quality defect). For example, for the repetition rate in the characterization quality index of the code data set, if there are repeated codes in the code data set (that is, the repetition rate is greater than 0), the corresponding repeated part in the code data set is identified as a quality defect of the code data set.

[0075] In this way, quality defects can be accurately identified, thereby facilitating targeted repairs and improving the quality of the code dataset, thereby enhancing the understanding and processing capabilities of the large language model trained or adjusted on the code dataset.

[0076] Step 400: Perform automatic or semi-automatic repair processing on the quality defects.

[0077] Specifically, the quality defects determined in step 300 are repaired automatically or semi-automatically. The automated repair can utilize technologies such as rule engines and machine learning models, and the semi-automated repair can utilize methods such as user interfaces to allow users to participate in repair decisions. In this way, compared with directly repairing quality defects manually, this method can effectively improve repair efficiency, reduce labor costs, and ensure consistency and accuracy of quality defect repair by automatically or semi-automatically repairing quality defects.

[0078] Step 500: Generate display information about quality defects and comparison information before and after repair of the quality defects.

[0079] Specifically, the corresponding information before and after the quality defect repair of the code data set is compared to obtain the comparison information before and after the quality defect repair (including the specific samples that have changed before and after the repair, the modification status and the changes in the corresponding quality indicators, etc.), and the display information about the quality defect and the comparison information before and after the quality defect repair is generated. The corresponding quality defect and the comparison information before and after the quality defect repair can be displayed on the human-computer interaction interface of the corresponding device, such as displayed on the corresponding display screen (as the human-computer interaction interface) of the corresponding device in a form that can be understood by the user (such as text, image, etc.), so that the user can clearly and intuitively understand the specific information of the corresponding quality defect, the effect of the quality defect repair and the improvement of the quality of the code data set, etc., which helps to improve the user's confidence and understanding of the repair operation and the repaired code data set.

[0080] For ease of understanding, the following is an example of the comparison information before and after the quality defect is repaired. Assuming that there are two types of defects in the code data set: high repetition rate and low format standard rate, the modification of the comparison information and the change of the corresponding quality indicators can be shown as follows:

[0081] “Before repair:

[0082] Duplication rate: 10% of the code samples are repeated.

[0083] Format standard rate: 15% of the samples do not meet the code format standards.

[0084] After the fix:

[0085] Repetition rate: Reduce the repetition rate to 0 through repair processing.

[0086] Format Standard Rate: Through repair processing, the number of samples that do not meet the format standard is reduced to 0. "

[0087] In this way, the method can realize the quality analysis and defect repair of the code data set used for training or adjusting (or fine-tuning) a large language model, and can be used to provide a high-quality code data set for training or adjusting a large language model. Specifically, by determining the representation quality index and content quality index of the code data set used for training or adjusting a large language model, the quality of the code data set is evaluated from multiple dimensions (or angles), the standardization (systematicity), objectivity and comprehensiveness of the quality evaluation of the code data set are guaranteed, and the applicability of the method is improved. According to the representation quality index and content quality index, the defects of the code data set are determined so as to accurately identify the quality defects, so as to facilitate targeted repair; by performing automated or semi-automated repair processing on the quality defects, the repair efficiency of the quality defects of the code data set is effectively improved, the labor cost is reduced, and the consistency and accuracy of the quality defect repair are guaranteed. By generating display information about quality defects and comparison information before and after the quality defect repair, so that the user can clearly and intuitively understand the specific information of the corresponding quality defects, the effect of the quality defect repair and the improvement of the quality of the code data set, it is helpful to improve the user's confidence and understanding of the repair operation and the repaired code data set.

[0088] Optionally, the characterization quality index includes at least one of a repetition rate, a format standard rate, a sample defect rate, a code syntax correctness rate, and a code block standard rate.

[0089] Specifically, in order to improve the comprehensiveness of the quality evaluation of the code data set, the quality indicators used by the method to evaluate the quality of the code data set include at least one of the repetition rate, format standard rate, sample defect rate, code syntax correctness rate and code block standard rate. That is, the quality indicators used by the method to evaluate the quality of the code data set include a combination of one or more of the repetition rate, format standard rate, sample defect rate, code syntax correctness rate and code block standard rate. Among them, the repetition rate is the proportion of similar or identical code samples in the code data set to the total code samples; high repetition rate as a corresponding quality defect may cause the large language model to learn too much about certain patterns, affecting its generalization ability. The format standard rate is the proportion of code samples in the code data set that meet the corresponding format standards (which can be determined according to the actual needs of the user) to the total code samples; code that meets the corresponding format standards is easier to understand and maintain, which helps to improve the readability of the code. The sample defect rate is the proportion of incomplete or missing code samples in the code data set to the total code samples; incomplete code samples may cause the large language model to learn incomplete grammatical structures, affecting the training or adjustment of the large language model. The code grammar correctness rate refers to the proportion of code samples that meet the corresponding grammar rules (which can be determined according to the actual needs of users) in the code data set to the total code samples; a high grammar correctness rate helps large language models learn the correct code structure and grammar. The code block standard rate refers to the proportion of code block samples that meet the corresponding standards (which can be determined according to the actual needs of users) in the code data set to the total code block samples; code blocks that meet the standards are easier to maintain and reuse, which helps improve the quality of the code.

[0090] In this way, this method is based on the representation quality index, comprehensively considering the quality characteristics of the code data set, such as repeatability, standardization, completeness, grammatical correctness and standardization, so as to comprehensively evaluate the quality of the code data set, and provide a multi-angle quantitative basis for subsequent defect analysis and repair. Moreover, the use of quantitative indicators such as repetition rate, format standard rate, sample defect rate, code grammatical correctness rate and code block standard rate can provide an objective measure for the evaluation of the representation quality of the code data set, avoiding the uncertainty of subjective evaluation; and the digital results are easier to understand and compare, reducing the inconsistency introduced by subjective judgment.

[0091] Step 300 includes:

[0092] Determine a characterization quality defect of the code data set with respect to a characterization quality indicator; wherein the quality defect comprises a characterization quality defect.

[0093] Specifically, in step 300, according to the characterization quality indicators (such as repetition rate, format standard rate, sample defect rate, code syntax correctness rate and code block standard rate, etc.), the characterization quality defects of these indicators in the code data set are determined. For example, when the repetition rate in the characterization quality indicator does not meet the specification (or preset rules, which are set by the user according to actual needs), if the repetition rate is greater than a certain threshold, the characterization quality defect is the quality defect of the repeated sample corresponding to the repetition rate.

[0094] Optionally, for the determination (or detection) of the characterization quality index of the code data set in step 200, illustratively, the detection of the repetition rate can be performed by applying a hash algorithm to the code data set, converting the code samples in the code data set into hash values ​​of fixed length, and comparing the hash values ​​of all samples. If two or more samples have the same hash value, then they may be the same or similar code samples. In this way, the repeated code samples are judged and found and statistics are performed, and they are stored in the corresponding repetition rate database to determine the repetition rate index. The detection of the format standard rate can be performed by detecting the predefined standardized rules for the code data set; for example, the rules that there cannot be continuous carriage returns in the code data set and there cannot be spaces at the end of the question are predefined. Through tools such as regular expressions, the standardized rules are applied to each sample in the code data set for detection, and all samples that do not meet the standardized rules are stored in the corresponding non-standardized data set database, and statistical information is calculated to obtain the format standard rate. The detection of sample incomplete rate can be done by pre-defining incomplete sample rules, that is, defining what samples are considered incomplete, such as defining that there are null values ​​in Prompt and Response, or that the response in Response ends with "," ":" or ";" as incomplete samples; and using tools such as regular expressions to apply incomplete sample rules to each sample in the code dataset for detection, and store all incomplete samples that meet the incomplete sample rules in the corresponding incomplete sample database, and calculate statistical information to obtain the sample incomplete rate. The detection of code syntax correctness is to check whether the code samples in the code dataset are grammatically correct to ensure that the code dataset can provide correct code for model learning. The code syntax analyzer can be used to identify the language features of the code, and then the corresponding code syntax analyzer is called to construct its syntax tree. The syntax tree analysis is used to detect whether each code sample has syntax errors, etc. All code samples with syntax errors are stored in the corresponding incorrect syntax sample database, and statistical information is calculated to obtain the code syntax correctness (or code syntax error rate).

[0095] The detection of the code block standard rate is mainly to detect whether the code in the code data set is correctly included by the ``` symbol to form a correct code block, so as to ensure that the code output after model learning is output in a user-friendly form. Since the code data set is a mixture of natural language and code, it is very difficult to simply use rules to check the code block. Therefore, the detection can be achieved by adjusting (or fine-tuning) the large model. Specifically, all correct samples can be obtained from the code data set by regular matching, and then the ``` symbol can be removed by regular rules to form missing samples. By constructing positive and negative samples, a data set for training (or adjusting) code block detection is formed. Finally, a binary classification model is trained on the corresponding base model, and the accuracy of detecting code blocks in the code data set is improved by embedding the code block detection model. All samples that do not meet the standard (the standard is that they are correctly included by the ``` symbol) are stored in the corresponding code block incomplete data set database, and statistical information is calculated to obtain the code block standard rate.

[0096] Optionally, step 400 includes:

[0097] When there are quality defects regarding the duplication rate, duplicate samples in the code dataset are deleted;

[0098] When there are quality defects related to the format standard rate, regular expressions are used to repair the corresponding defective samples;

[0099] When there are quality defects regarding the sample defect rate, the defective samples in the code dataset are deleted;

[0100] When there are quality defects related to the code syntax correctness, the second preset model is used to repair the corresponding defective samples;

[0101] When there is a quality defect with respect to the standard rate of the code block, the third preset model is used to perform repair processing on the corresponding defective sample.

[0102] Repair the quality defects in the code data set regarding the characterization quality indicators (i.e., characterization quality defects). Specifically, when there is a characterization quality defect regarding the repetition rate, the defect is repaired by directly deleting the repeated samples. When there is a characterization quality defect regarding the format standard rate, the defective samples in the code data set are directly repaired by means of regular expressions. When there is a characterization quality defect regarding the sample defect rate, the defect is repaired by directly deleting the detected defective samples.

[0103] When there are quality defects that characterize the grammatical accuracy of the code, the second preset model is used to repair the corresponding defective samples; illustratively, the code with grammatical defects detected is first passed through the corresponding code expert model (i.e., the second preset model), and the code expert model is allowed to complete the grammatical defect determination task for the first time, and then another iteration is performed to send the grammatical problems and defective codes generated for the first time to the code expert model for repair; the repaired code is tested through the corresponding grammar detector, and if the test passes, the process is completed, otherwise a second repair is performed until the repaired code can pass the grammar detector. Among them, the maximum number of repair attempts N can be set to limit the number of iterations and improve the efficiency of automated repair; if the data cannot be repaired within N times, the N+1th manual repair operation is performed to achieve semi-automatic repair processing and complete the repair of data grammatical defects.

[0104] When there is a quality defect in characterizing the standard rate of a code block, a third preset model is used to repair the corresponding defective code block. The third preset model can be obtained by adjusting (or fine-tuning) the corresponding large language model as the base model. Among them, the defective code block can be obtained by subjecting the original correct code block sample to some changes, damage or artificial introduction of errors, and the defective code block is used as input and the original correct code is used as output for post-training adjustment (or fine-tuning) of the third preset model. The base model can use a sequence model to ensure that the obtained third preset model can automatically repair the corresponding defective samples detected.

[0105] Optionally, the content quality indicator includes at least one of scores of the code dataset regarding intent completeness, response correctness, code logic correctness, and question-answer correspondence.

[0106] Specifically, in order to improve the comprehensiveness of the quality evaluation of the code data set, the content quality index used by the method to evaluate the quality of the code data set includes at least one of the scores of the code data set on the completeness of intent, the correctness of response, the correctness of code logic, and the correspondence between questions and answers. That is, the content quality of the code data set can be evaluated (scored) from at least one of the aspects of completeness of intent, correctness of response, correctness of code logic, and correspondence between questions and answers. Among them, the completeness of intent is used to measure whether the code snippets in the code data set express the complete programming intent, that is, to evaluate whether the code snippets contain enough information to accurately understand the programming purpose; regarding the scoring of the completeness of intent, the higher the score, the more code snippets with complete intent expression in the code data set (or the higher the proportion of total code snippets). Response correctness is used to measure whether the code snippets respond to specific inputs or conditions in the correct way, which helps the large language model identify code that may cause problems in specific situations; regarding the scoring of response correctness, the higher the score, the more code snippets in the code data set respond to specific inputs or conditions in the correct way (or the higher the proportion of total code snippets). The code logic correctness is used to measure whether the logic of the code snippets in the code dataset is logically standardized, which helps the large language model evaluate the structure and flow of the code to ensure its logical correctness; regarding the scoring of code logic correctness, the higher the score, the more code snippets in the code dataset are logically standardized (or the higher the proportion of total code snippets). Question-answer correspondence is used to measure whether the code snippets involved in interaction or question answering correctly answer the questions, which helps ensure that the code can accurately respond to questions raised by users in actual application scenarios; regarding the scoring of question-answer correspondence, the higher the score, the more code snippets in the code dataset correctly answer the questions (or the higher the proportion of total code snippets involved in interaction or question answering).

[0107] In this way, this method is based on content quality indicators and comprehensively considers multiple quality characteristics of the code dataset, such as intent completeness, response correctness, code logic correctness, and question-answer correspondence, to comprehensively evaluate the quality of the code dataset and provide a multi-angle quantitative basis for subsequent defect analysis and repair.

[0108] Optionally, determining the representation quality index and content quality index of the code data set includes:

[0109] A first preset expert model is used to adjust the code data set with respect to intent completeness to obtain a first code adjustment data set; the code data set and the first code adjustment data set are input into a preset universal scoring model, and a first score of the code data set with respect to intent completeness is obtained as an output; and a score of the code data set with respect to intent completeness is determined based on the first score obtained at least once.

[0110] And / or, using a second preset expert model to adjust the code dataset with respect to response correctness to obtain a second code adjustment dataset; inputting the code dataset and the second code adjustment dataset into a preset universal scoring model, and outputting a second score of the code dataset with respect to response correctness; determining the score of the code dataset with respect to response correctness based on the second score obtained at least once;

[0111] And / or, using a third preset expert model to adjust the code data set with respect to the code logic correctness, to obtain a third code adjustment data set; inputting the code data set and the third code adjustment data set into a preset universal scoring model, and outputting a third score of the code data set with respect to the code logic correctness; determining the score of the code data set with respect to the code logic correctness according to the third score obtained at least once;

[0112] And / or, a fourth preset expert model is used to adjust the code dataset with respect to question-answer correspondence to obtain a fourth code adjusted dataset; the code dataset and the fourth code adjusted dataset are input into a preset universal scoring model, and a fourth score of the code dataset with respect to question-answer correspondence is outputted; and the score of the code dataset with respect to question-answer correspondence is determined based on the fourth score obtained at least once.

[0113] This method uses a pre-trained model for corresponding scoring (referred to as the first preset model) to evaluate the content quality indicators of the code dataset, such as the completeness of intent, correctness of response, correctness of code logic, and correspondence between questions and answers, by scoring. In this way, a more objective and standardized evaluation can be provided, the interference of subjective factors in quality evaluation can be avoided, and the efficiency of evaluating the code dataset can be improved. Moreover, quantitative scoring can provide a more precise and accurate measure for the evaluation of the content quality of the code dataset, reducing errors introduced by human subjective judgment.

[0114] If it is necessary to determine the score of the code data set with respect to the completeness of intent. Specifically, first, through step a, the first preset expert model is used to fine-tune (or adjust) the code data set with respect to the completeness of intent, that is, the code snippets (samples) with incomplete intent expression in the code data set are adjusted to make the intent expression complete, and the first code adjustment data set is obtained after the code adjustment data set is adjusted. Thereafter, through step b, the code data set is given a first score with respect to the completeness of intent by using a preset general scoring model. For example, the code data set and the first code adjustment data set are input into the preset general scoring model, and the preset general scoring model is used to perform a comparative analysis based on the input original code data set and the first code adjustment data set, such as taking the first code adjustment data set as the full score standard with respect to the completeness of intent, and determining the first score of the code data set with respect to the completeness of intent according to the degree of similarity between the original code data set and the first code adjustment data set. Repeating steps a to b for a first preset number of times (which is an integer greater than or equal to 0), and counting the first scores obtained by each execution of steps a to b, a number of first scores equal to the first preset number of times can be obtained, and the score of the code data set with respect to the completeness of intent can be determined according to all the first scores obtained at least once.

[0115] If it is necessary to determine the score of the code data set regarding the correctness of the response. Specifically, first, through step c, the code data set is adjusted regarding the correctness of the response using the second preset expert model, that is, the code fragments in the code data set that do not respond to specific inputs or conditions in a correct manner are adjusted so that they can respond to specific inputs or conditions in a correct manner, and the second code adjustment data set is adjusted to obtain the second code adjustment data set. Thereafter, through step d, the code data set is scored for the second correctness of the response using the preset universal scoring model. For example, the code data set and the second code adjustment data set are input into the preset universal scoring model. Through the preset universal scoring model, a comparative analysis is performed based on the input original code data set and the second code adjustment data set. For example, the second code adjustment data set is used as the full score standard for the correctness of the response, and the second score of the code data set regarding the correctness of the response is determined based on the degree of similarity between the original code data set and the second code adjustment data set. Repeat steps c to d for a first preset number of times, and count the second scores obtained each time steps c to d are executed. The number of second scores equal to the first preset number of times can be obtained. Based on all the second scores obtained at least once, the score of the code data set regarding the correctness of the response can be determined.

[0116] If it is necessary to determine the score of the code data set regarding the correctness of the code logic. Specifically, first, through step e, the third preset expert model is used to adjust the code data set regarding the correctness of the code logic, that is, the code fragments that are not in line with the logic standard in the code data set are adjusted to make their logic in line with the logic standard, and the third code adjustment data set is obtained after the adjustment of the code adjustment data set. Thereafter, through step f, the code data set is given a third score regarding the correctness of the code logic by using a preset universal scoring model. For example, the code data set and the third code adjustment data set are input into the preset universal scoring model, and the preset universal scoring model is used to perform a comparative analysis based on the input original code data set and the third code adjustment data set. For example, the third code adjustment data set is used as the full score standard for the correctness of the code logic, and the third score of the code data set regarding the correctness of the code logic is determined according to the degree of similarity between the original code data set and the third code adjustment data set. Repeat steps e to f for a first preset number of times, and count the third scores obtained by each execution of steps e to f, so as to obtain a number of third scores equal to the first preset number of times. According to all the third scores obtained at least once, the score of the code data set regarding the correctness of the code logic can be determined.

[0117] If it is necessary to determine the score of the code data set on the correspondence between questions and answers. Specifically, first, through step g, the fourth preset expert model is used to adjust the code data set on the correspondence between questions and answers, that is, the code snippets that the code data set does not correctly answer the corresponding questions are adjusted so that they can correctly (or reasonably) answer the corresponding questions. After the code adjustment data set is adjusted, the fourth code adjustment data set is obtained. After that, through step h, the code data set is given a fourth score on the correspondence between questions and answers by using a preset universal scoring model. For example, the code data set and the fourth code adjustment data set are input into the preset universal scoring model. Through the preset universal scoring model, a comparative analysis is performed based on the input original code data set and the fourth code adjustment data set. For example, the fourth code adjustment data set is used as the full score standard for the correspondence between questions and answers, and the fourth score of the code data set on the correspondence between questions and answers is determined according to the degree of similarity between the original code data set and the fourth code adjustment data set. Repeat steps g to h for a first preset number of times, and count the fourth scores obtained by each execution of steps g to h. The fourth scores equal to the first preset number of times can be obtained. According to all the fourth scores obtained at least once, the score of the code data set on the correspondence between questions and answers can be determined.

[0118] In this way, based on comparative analysis, the objectivity and reliability of the scoring results are guaranteed. In order to avoid the contingency of a single scoring, the corresponding steps of obtaining the corresponding scores (such as the first score, the second score, the third score, and the fourth score) are repeated to obtain multiple corresponding scores, and the final score is determined based on the multiple corresponding scores obtained, so that multiple iterations of the scoring are achieved, more randomness is introduced in the multiple iterations, and the uncertainty caused by the contingency of a single scoring is reduced, thereby improving the reliability and comprehensiveness of the scoring.

[0119] Optionally, determining the score of the code dataset regarding the intent completeness according to the at least one first score obtained comprises:

[0120] A score of the code dataset with respect to intent completeness is determined based on a first variance and a first average of all first scores.

[0121] Specifically, the number of all first scores obtained is at least one. When the number of first scores is multiple, the degree of dispersion of all first scores can be measured by calculating the variance of all first scores (recorded as the first variance), that is, the measure of the difference between all first scores and the average value of all first scores (recorded as the first average value), which is also a measure of the stability of the first scores. Based on this, the final code data set score on intent completeness is determined. Obviously, if the first scores on intent completeness obtained multiple times are similar, it means that the reliability and stability of the obtained first scores are high, and can be used to obtain the final score on intent completeness; and if the first scores on intent completeness obtained multiple times are more dispersed, that is, the corresponding variance is larger, it means that the first score fluctuates greatly in multiple iterations and is not stable enough. In this case, it may be necessary to further explore the reasons for the score fluctuations and consider increasing the number of iterations or manual access to score.

[0122] And / or, determining the score of the code dataset regarding the correctness of the response based on the at least one second score obtained comprises:

[0123] A score of the code data set regarding response correctness is determined based on a second variance and a second mean of all second scores.

[0124] Specifically, the number of all the second scores obtained is at least one. When the number of the second scores is multiple, the degree of dispersion of all the second scores can be measured by obtaining the variance of all the second scores (recorded as the second variance), that is, the measurement value of the difference between all the second scores and the average value of all the second scores (recorded as the second average value), which is also a measure of the stability of the second scores. Based on this, the score of the final code data set on the correctness of the response is determined. Obviously, if the second scores on the correctness of the response obtained multiple times are similar, it means that the reliability and stability of the obtained second scores are high, and can be used to obtain the final score on the correctness of the response; if the second scores on the correctness of the response obtained multiple times are dispersed to a large extent, that is, the corresponding variance is large, it means that the second scores fluctuate greatly in multiple iterations and are not stable enough. In this case, it may be necessary to further explore the reasons for the fluctuation of the scores, and consider increasing the number of iterations or manual access to score.

[0125] And / or, determining the score of the code dataset regarding the logic correctness of the code according to the third score obtained at least once comprises:

[0126] The score of the code data set regarding the logical correctness of the code is determined according to the third party differences and the third average of all the third scores.

[0127] Specifically, the number of all the third scores obtained is at least one. When the number of the third scores is multiple, the degree of dispersion of all the third scores can be measured by calculating the variance of all the third scores (recorded as the third variance), that is, the measurement value of the difference between all the third scores and the average value of all the third scores (recorded as the third average value), which is also a measure of the stability of the third scores. Based on this, the final code data set score on the integrity of intent is determined. Obviously, if the third scores on the logical correctness of the code obtained multiple times are similar, it means that the reliability and stability of the obtained third scores are high, and can be used to obtain the final score on the logical correctness of the code; and if the third scores on the logical correctness of the code obtained multiple times are more dispersed, that is, the corresponding variance is larger, it means that the third scores fluctuate greatly in multiple iterations and are not stable enough. In this case, it may be necessary to further explore the reasons for the fluctuation of the scores, and consider increasing the number of iterations or manual access to score.

[0128] And / or, determining the score of the code dataset regarding the correspondence between the question and the answer based on the fourth score obtained at least once includes:

[0129] A score of the code dataset with respect to question-answer correspondence is determined according to a fourth variance and a fourth mean of all fourth scores.

[0130] Specifically, the number of all fourth scores obtained is at least one. When the number of fourth scores is multiple, the degree of dispersion of all fourth scores can be measured by calculating the variance of all fourth scores (recorded as fourth variance), that is, the measure of the difference between all fourth scores and the average value of all fourth scores (recorded as fourth average value), which is also a measure of the stability of the fourth scores. Based on this, the final code data set score on intent completeness is determined. Obviously, if the fourth scores on question-answer correspondence obtained multiple times are similar, it means that the reliability and stability of the obtained fourth scores are relatively high, and can be used to obtain the final score on question-answer correspondence; and if the fourth scores on question-answer correspondence obtained multiple times are dispersed to a large extent, that is, the corresponding variance is large, it means that the fourth scores fluctuate greatly in multiple iterations and are not stable enough. In this case, it may be necessary to further explore the reasons for the score fluctuations and consider increasing the number of iterations or manual access to score.

[0131] Optionally, determining the score of the code dataset regarding the intent completeness according to the first variance and the first average of all the first scores includes:

[0132] When the first variance is less than or equal to a first preset threshold, the first average value is used as the score of the code dataset regarding intent completeness; when the first variance is greater than the first preset threshold, a prompt message for manual scoring of intent completeness is generated, and in response to receiving the first manual score of intent completeness, the first manual score is used as the score of the code dataset regarding intent completeness.

[0133] And / or, determining the score of the code dataset regarding the correctness of the response according to the second variance and the second mean of all the second scores comprises:

[0134] When the second variance is less than or equal to the second preset threshold, the second average value is used as the score of the code data set regarding the response correctness; when the second variance is greater than the second preset threshold, a prompt message for manual scoring of the response correctness is generated, and in response to receiving the second manual score regarding the response correctness, the second manual score is used as the score of the code data set regarding the response correctness;

[0135] And / or, determining the score of the code data set regarding the logical correctness of the code according to the third party differences and the third averages of all the third scores comprises:

[0136] When the third difference is less than or equal to a third preset threshold, the third average value is used as the score of the code data set regarding the code logic correctness; when the third difference is greater than the third preset threshold, a prompt message for manual scoring of the code logic correctness is generated, and in response to receiving the third manual score regarding the code logic correctness, the third manual score is used as the score of the code data set regarding the code logic correctness;

[0137] And / or, determining the score of the code dataset regarding the correspondence between the question and the answer based on the fourth variance and the fourth mean of all the fourth scores includes:

[0138] When the fourth variance is less than or equal to the fourth preset threshold, the fourth average value is used as the score of the code dataset regarding the correctness of the response; when the fourth variance is greater than the fourth preset threshold, a prompt message is generated regarding performing a manual score on the question-answer correspondence, and in response to receiving the fourth manual score on the question-answer correspondence, the fourth manual score is used as the score of the code dataset regarding the question-answer correspondence.

[0139] Specifically, the calculated corresponding variance is compared with the corresponding preset threshold value, which is a reference value set during design to judge the stability of the score.

[0140] If the corresponding variance is less than or equal to the corresponding preset threshold, it means that the score on this dimension (or indicator) is relatively stable and not affected by too many fluctuations. The average of the first scores on a certain dimension (or indicator) obtained multiple times (i.e., the first preset number plus one time) can be used as the score of the code data set on this dimension (or indicator). For example, if the first variance is less than or equal to the first preset threshold, the average value of at least one or all first scores on the completeness of intent (recorded as the first average value) is used as the score of the code data set on the completeness of intent; if the second variance is less than or equal to the second preset threshold, the average value of at least one or all first scores on the correctness of response (recorded as the second average value) is used as the score of the code data set on the correctness of response; if the third variance is less than or equal to the third preset threshold, the average value of at least one or all first scores on the correctness of code logic (recorded as the third average value) is used as the score of the code data set on the correctness of code logic; if the fourth variance is less than or equal to the fourth preset threshold, the average value of at least one or all first scores on the correspondence between question and answer (recorded as the fourth average value) is used as the score of the code data set on the correctness of response. In this way, automatic scoring is achieved.

[0141] If the corresponding variance is greater than the corresponding preset threshold, it means that the score on this dimension (or indicator) is not stable enough, has a large degree of dispersion, and has a large fluctuation. In this case, it may be necessary to further explore the reasons for the fluctuation of the score, and consider increasing the number of iterations or manual access to score. For manual access to score (which belongs to semi-automatic scoring, which is different from the above-mentioned automatic scoring), specifically, when the corresponding variance is greater than the corresponding preset threshold, a prompt message for manual scoring of the corresponding indicator is generated to prompt the user (or other relevant personnel) to manually score the code data set for the corresponding indicator; after receiving the manual score for the corresponding indicator, the manual score is used as the score of the code data set for the indicator. For example, if the first variance is greater than the first preset threshold, a prompt message for manual scoring of the intent completeness is generated; in response to receiving the first manual score for the intent completeness, the first manual score is used as the score of the code data set for the intent completeness. If the second variance is greater than the second preset threshold, a prompt message for manual scoring of the response correctness is generated; in response to receiving the second manual score for the response correctness, the second manual score is used as the score of the code data set for the response correctness. If the third variance is greater than the third preset threshold, a prompt message for manual scoring of the code logic correctness is generated; in response to receiving the third manual scoring of the code logic correctness, the third manual scoring is used as the scoring of the code data set on the code logic correctness. If the fourth variance is greater than the fourth preset threshold, a prompt message for manual scoring of the question-answer correspondence is generated; in response to receiving the fourth manual scoring of the question-answer correspondence, the fourth manual scoring is used as the scoring of the code data set on the question-answer correspondence.

[0142] Exemplarily, for scoring the code data set with respect to the completeness of intent, correctness of response, correctness of code logic, and correspondence between questions and answers, in this method, the code data set is first input into the first preset expert model, the second preset expert model, the third preset expert model, and the fourth preset expert model, respectively, and the four preset expert models are allowed to adjust the code data set with respect to the completeness of intent, correctness of response, correctness of code logic, and correspondence between questions and answers, respectively, to obtain the adjusted first code adjustment data set, the second code adjustment data set, the third code adjustment data set, and the fourth code adjustment data set. Then, the original code data set (i.e., the code data set before adjustment) and the adjusted data obtained by the output of the four preset expert models (i.e., the first code adjustment data set, the second code adjustment data set, the third code adjustment data set, and the fourth code adjustment data set) are input into the preset general scoring model for quantitative scoring with respect to the completeness of intent, correctness of response, correctness of code logic, and correspondence between questions and answers, such as scoring values ​​of 0-10 points. Then, the first score, the second score, the third score and the fourth score are obtained repeatedly. After repeating the first preset number of iterations (if the first preset number is 2, it is performed three times in total, the first time plus the two subsequent repetitions), the mean u and variance σ of the scores of the three evaluations of different indicators are counted; if the variance σ of the three evaluations of the corresponding indicator is greater than the preset corresponding threshold, four manual judgments are made to give the final score; otherwise, the mean of the three scores will be used as the corresponding score of the indicator. Among them, it can also be that four preset expert models evaluate the code data set on the completeness of intent, correctness of response, correctness of code logic and correspondence between questions and answers, and give their corresponding suggestions (such as modification suggestions), and then the original code data set and the corresponding evaluation and suggestions are input into the preset general scoring model, and the code data set is scored on the completeness of intent, correctness of response, correctness of code logic and correspondence between questions and answers.

[0143] It is worth noting that the first preset threshold, the second preset threshold, the third preset threshold, and the fourth preset threshold can be set according to the actual needs of the user, and can take the same value or different values.

[0144] Step 300 includes:

[0145] Determine content quality defects of the code data set with respect to a content quality indicator; wherein the quality defects include content quality defects.

[0146] Specifically, in step 300, based on content quality indicators (such as scores for intent completeness, response correctness, code logic correctness, and question-answer correspondence, etc.), content quality defects in the code dataset with respect to these indicators are determined. For example, if the intent completeness in the content quality indicator does not meet the specification, that is, the score for intent completeness is lower than the corresponding threshold (which is set according to the actual needs of the user), the content quality defect is a quality defect of the code sample with incomplete intent expression in the code dataset.

[0147] Optionally, determining content quality defects of the code dataset with respect to a content quality indicator includes:

[0148] The content quality defects of the code dataset are determined based on the scores of the code dataset on intent completeness, response correctness, code logic correctness, and question-answer correspondence.

[0149] Specifically, according to the scores of the code data set on the completeness of intent, correctness of response, correctness of code logic and correspondence of question and answer and the corresponding preset thresholds, a comparison is made to determine that the dimensions corresponding to the scores below the threshold (i.e., completeness of intent, correctness of response, correctness of code logic and correspondence of question and answer) have corresponding content quality defects. For example, if the score range for completeness of intent is 0-10 points and the corresponding threshold is set to 8 points, it is considered that the code data set with a score of not less than 8 points for completeness of intent is of high quality in terms of completeness of intent, and the code data set with a score of less than 8 points for completeness of intent has content quality defects in terms of completeness of intent.

[0150] Optionally, for using the first preset model to score the code data set respectively on the intent completeness, response correctness, code logic correctness and question-answer correspondence, the content quality evaluation of the code data set is realized by performing multiple rounds of voting (scoring) of large-model experts through the first preset expert model, the second preset expert model, the third preset expert model, the fourth preset expert model and the preset general scoring model. Among them, the first preset expert model can adopt the Prompt expert model that performs reinforcement learning on the Prompt instruction judgment, which focuses on scoring the intent completeness in the code data set; the second preset expert model and the fourth preset expert model can both adopt the Response expert model that performs reinforcement learning on the Prompt corresponding to the Response, the second preset expert model focuses on scoring the response correctness in the code data set, and the fourth preset expert model focuses on scoring the question-answer correspondence in the code data set; the third preset expert model can adopt the code expert model that performs reinforcement learning on the code data, which focuses on scoring the code logic correctness in the code data set; the preset general scoring model can adopt the scoring model learned on the general task, which is used to perform comprehensive quantitative scoring on the code data set.

[0151] Optionally, combined Figure 1 , Figure 2 As shown, step 400 includes:

[0152] Step 410: When the code data set has content quality defects regarding at least one of intent completeness, response correctness, code logic correctness, and question-answer correspondence, a first defect description and repair suggestion information regarding the content quality defect is determined through a fifth preset model.

[0153] Specifically, when there is at least one of content quality defects regarding intent completeness, content quality defects regarding response correctness, content quality defects regarding code logic correctness, and content quality defects regarding question-answer correspondence in the code dataset, a preset model (recorded as the fifth preset model) that focuses on identifying corresponding defects in the code dataset is adopted to obtain relevant information about defect descriptions and repair suggestions for the corresponding content quality defects (recorded as the first defect description and repair suggestion information) for determining the quality defect problems of the code dataset and their repair directions.

[0154] Step 420: According to the first defect description and the repair suggestion information, a sixth preset model is used to repair the code data set to obtain a corrected code data set.

[0155] Specifically, based on the first defect description and repair suggestion information, a preset corresponding model (referred to as the sixth preset model) is used to repair the code dataset to obtain a repaired code dataset (referred to as the corrected code dataset). The sixth preset model focuses on code repair and can automatically correct the code through the provided description and suggestions.

[0156] Step 430: Use the first preset model to score the corrected code data set to obtain a second content quality score corresponding to the corrected code data set, wherein the first preset model includes a first preset expert model, a second preset expert model, a third preset expert model, a fourth preset expert model and a preset general scoring model.

[0157] Specifically, the above-mentioned first preset model is used to score the repaired revised code dataset, such as using the above-mentioned first preset model including the first preset expert model, the second preset expert model, the third preset expert model, the fourth preset expert model and the preset general scoring model, to score the revised code dataset with respect to corresponding indicators (such as intent completeness, response correctness, code logic correctness or question-answer correspondence), and the corresponding score obtained is recorded as the second content quality score, which is used to determine the effect of the current repair, or to evaluate the repaired code dataset in terms of intent completeness, response correctness, code logic correctness or question-answer correspondence. In some embodiments, the type of scoring of the revised code dataset in step 430 (such as the first score corresponding to intent completeness, the second score corresponding to response correctness, the third score corresponding to code logic correctness, and the fourth score corresponding to question-answer correspondence) corresponds to the corresponding content quality defects existing in the code dataset in step 410.

[0158] Step 440: If the second content quality score is higher than the first content quality score corresponding to the code data set and higher than the fifth preset threshold, determine that the corresponding content quality defect is repaired; if the second content quality score is lower than at least one of the first content quality score and the fifth preset threshold, use the corrected code data set as the code data set in step 410, and repeat steps 410 to 440 until the second content quality score corresponding to the obtained corrected code data set is higher than the first content quality score and higher than the fifth preset threshold; wherein the first content quality score includes at least one of the scores of the code data set regarding intent completeness, response correctness, code logic correctness, and question-answer correspondence.

[0159] Specifically, if the second content quality score corresponding to the repaired corrected code dataset is higher than the first content quality score corresponding to the original code dataset (which includes at least one of the scores of the code dataset regarding intent completeness, response correctness, code logic correctness, and question-answer correspondence, and is of the same type as the second content quality score, so as to compare the scores of the same type), and is higher than the fifth preset threshold, it is determined that the corresponding content quality defect has been repaired, which means that the repaired corrected code dataset has been improved relative to the code dataset in terms of the corresponding indicators of content quality. Among them, the fifth preset threshold can be set according to the actual needs of the user, which is used to determine the quality of the code dataset in terms of the corresponding indicators of content quality. Generally, if the first content quality score corresponding to the corresponding indicator is not lower than the fifth preset threshold, it means that the code dataset is of high quality in terms of the corresponding indicator; if the first content quality score corresponding to the corresponding indicator is lower than the fifth preset threshold, it means that the code dataset has content quality defects in terms of the corresponding indicator.

[0160] If the second content quality score corresponding to the corrected code data set obtained after repair is lower than at least one of the first content quality score corresponding to the original code data set and the fifth preset threshold (wherein, by default, the first content quality score corresponding to the original code data set with corresponding content quality defects is lower than the fifth preset threshold), then it can be determined that the corrected code data set after repair still has corresponding content quality defects, and the corrected code data set needs to be repaired again. Specifically, the repaired corrected code data set is used as the code data set in step 410, and steps 410 to 440 are repeatedly executed until the second content quality score corresponding to the corrected code data set obtained again is higher than the first content quality score corresponding to the code data set, and higher than the fifth preset threshold, that is, a code data set with high quality in terms of corresponding indicators is obtained, and the repair (quality improvement) of the corresponding indicators of the original code data set is completed.

[0161] For ease of understanding, the following description is made by taking the case where a code dataset has content quality defects regarding intent integrity as an example. Then step 400 includes:

[0162] Step 410: When the code data set has a content quality defect related to the intent integrity, determine a first defect description and repair suggestion information related to the content quality defect related to the intent integrity through a fifth preset model;

[0163] Step 420: According to the first defect description and the repair suggestion information, a sixth preset model is used to repair the code data set to obtain a corrected code data set;

[0164] Step 430: Use the first preset model to score the corrected code dataset regarding the completeness of intent, and obtain a score of the corrected code dataset regarding the completeness of intent (a second content quality score);

[0165] Step 440: If the score of the corrected code dataset regarding intent completeness is higher than the score of the code dataset regarding intent completeness and higher than the fifth preset threshold, it is determined that the repair of the content quality defects of the code dataset regarding intent completeness is completed; if the score of the corrected code dataset regarding intent completeness is lower than at least one of the score of the code dataset regarding intent completeness and the fifth preset threshold, it is determined that the content quality defects regarding intent completeness still exist in the corrected code dataset, and the corrected code dataset is used as the code dataset in step 410, and steps 410 to 440 are repeatedly performed until the score of the corrected code dataset regarding intent completeness is higher than the score of the code dataset regarding intent completeness and higher than the fifth preset threshold, that is, a code dataset of high quality in terms of intent completeness is obtained, and the repair (quality improvement) of the original code dataset in terms of intent completeness is completed.

[0166] Optionally, the second content quality score corresponding to the revised code dataset includes a first score of the revised code dataset regarding intent completeness, a second score regarding response correctness, a third score regarding code logic correctness, and a fourth score regarding question-answer correspondence; in some embodiments, the second content quality score of the revised code dataset also includes an average of multiple first scores, an average of multiple second scores, an average of multiple third scores, and an average of multiple fourth scores.

[0167] Optionally, the first content quality score corresponding to the code dataset includes a first score of the code dataset regarding intent completeness, a second score regarding response correctness, a third score regarding code logic correctness, and a fourth score regarding question-answer correspondence, as well as a score of the code dataset regarding intent completeness (final score) determined based on the first score obtained at least once, a score of the code dataset regarding response correctness (final score) determined based on the second score obtained at least once, a score of the code dataset regarding code logic correctness (final score) determined based on the third score obtained at least once, and a score of the code dataset regarding question-answer correspondence (final score) determined based on the fourth score obtained at least once.

[0168] Optionally, for the repair process of content quality defects, in order to further improve its repair efficiency, the maximum number of repair attempts M can be set to limit the number of iterations (i.e., the number of times steps 410 to 440 are repeated). After reaching M times, if the second content quality score corresponding to the corrected code data set is still lower than the first content quality score corresponding to the code data set and at least one of the fifth preset thresholds, repair processing is performed through manual intervention to achieve semi-automatic repair processing of the corresponding defects and improve repair efficiency.

[0169] Optionally, after step 100 and before step 400, the method further includes:

[0170] A preset statistical algorithm is used to obtain the statistical characteristics of the code data set.

[0171] Specifically, in order to facilitate the user to clearly understand the code data set obtained in step 100, a preset statistical algorithm (referred to as a preset statistical algorithm) is used to obtain the statistical features of the code data set (which is an indicator for quantitatively describing the samples or attributes in the code data set, such as data volume, task type, code language type, code defect statistics, sample length statistics, code complexity, etc.), such as obtaining the key features of the code data set through a statistical algorithm, so as to fully understand the distribution and trend of the data and control the data quality. Among them, the preset statistical algorithm can be a variety of algorithms for data analysis and statistics. In some embodiments, it is also possible to call the corresponding statistical algorithm according to the features selected by the user to calculate and obtain the corresponding statistical features in the code data set; that is, the corresponding features and statistical algorithms can be designed and customized by the user, and can be selected by the user according to the characteristics of the code data set. In other embodiments, the statistical features include total data features of the data set, data length statistical features, data set task classification quantity statistical features, data set language feature statistical information and data set code language statistical features; wherein, the data set task classification quantity statistical features, data set language feature statistical information and data set code language statistical features are provided by the code data set The corresponding label information of the corresponding sample.

[0172] Step 400 includes:

[0173] Generates a display of statistical information about features.

[0174] Specifically, by converting statistical features into easy-to-understand display information, such as charts, graphs or summary text, and displaying them on the corresponding display screen of the corresponding device, visual display is achieved to help users understand the statistical properties of the code data set more clearly and intuitively, and to facilitate users to find corresponding information and count corresponding data.

[0175] Optionally, the display information about statistical characteristics and the display information about quality defects and comparative information before and after the quality defects are repaired can be displayed together on the corresponding display screens of the same device, or can be displayed on the corresponding display screens of different devices, or can be displayed one of the display information about statistical characteristics and the display information about quality defects and comparative information before and after the quality defects are repaired on the corresponding display screen of the same device.

[0176] Combination Figure 3 As shown, another embodiment of the present invention provides a quality analysis and defect repair device for a code data set, comprising:

[0177] An acquisition unit, used to acquire a code dataset for model training or adjustment;

[0178] A quality indicator determination unit, used to determine a representation quality indicator and a content quality indicator of a code data set, wherein the representation quality indicator is used to refer to the quality of a data presentation form of the code data set, and the content quality indicator is used to refer to the quality of a data presentation content of the code data set;

[0179] A quality defect determination unit, used for determining quality defects of a code data set according to a characterization quality indicator and a content quality indicator;

[0180] Quality defect repair unit, used to automatically or semi-automatically repair quality defects;

[0181] The display unit is used to generate display information about the quality defect and the comparison information before and after the quality defect is repaired.

[0182] In this embodiment, through the mutual cooperation between the acquisition unit, quality indicator determination unit, quality defect determination unit, quality defect repair unit, display unit and other structures of the quality analysis and defect repair device of the code data set, the quality analysis and defect repair method of the code data set can be smoothly and stably performed, and the quality analysis and defect repair of the code data set used for training or fine-tuning of a large language model can be realized, which can be used to provide a high-quality code data set for training or fine-tuning of a large language model. Specifically, by determining the characterization quality index and content quality index of the code data set used for training or fine-tuning of a large language model, the quality of the code data set is evaluated from multiple dimensions (or angles), the standardization (systematic), objectivity and comprehensiveness of the quality evaluation of the code data set are guaranteed, and the applicability is improved. According to the characterization quality index and content quality index, the defects of the code data set are determined so as to accurately identify the quality defects, thereby facilitating targeted repair; by performing automated or semi-automated repair processing on the quality defects, the repair efficiency of the quality defects of the code data set can be effectively improved, the labor cost can be reduced, and the consistency and accuracy of the quality defect repair can be guaranteed. By generating display information about quality defects and comparative information before and after quality defect repair, users can clearly and intuitively understand the specific information of the corresponding quality defects, the effect of quality defect repair, and the improvement of the quality of the code data set, which helps to improve users' confidence and understanding of the repair operation and the repaired code data set.

[0183] Among them, when the quality defect repair unit is used to perform semi-automatic repair processing on quality defects, the quality defect repair unit can automatically execute corresponding steps, and perform corresponding repair processing through manual intervention, thereby realizing semi-automatic repair processing of corresponding defects.

[0184] Combination Figure 4 As shown, another embodiment of the present invention provides an electronic device, including a memory 401 and a processor 402;

[0185] Memory 401, used for storing computer programs;

[0186] The processor 402 is used to implement the above-mentioned quality analysis and defect repair method of the code data set when executing the computer program.

[0187] In other words, an electronic device includes a memory 401 and a processor 402 coupled to the memory 401; the memory 401 is configured to store a computer program; and the processor 402 is configured to perform the following operations when executing the computer program:

[0188] Get the code dataset for model training or tuning;

[0189] Determine the representation quality index and content quality index of the code dataset, wherein the representation quality index is used to refer to the quality of the data presentation form of the code dataset, and the content quality index is used to refer to the quality of the data presentation content of the code dataset;

[0190] Determine the quality defects of the code dataset based on representation quality indicators and content quality indicators;

[0191] Automatic or semi-automatic repair of quality defects;

[0192] Generates display information about quality defects and comparative information before and after the quality defects are repaired.

[0193] In this embodiment, through the cooperation of the processor 402, memory 401 and other structures of the electronic device, the quality analysis and defect repair method of the code data set is smoothly and stably executed, and the quality analysis and defect repair of the code data set used for large language model training or fine-tuning can be realized, which can be used to provide a high-quality code data set for the training or fine-tuning of the large language model. Specifically, by determining the characterization quality index and content quality index of the code data set used for large language model training or fine-tuning, the quality of the code data set is evaluated from multiple dimensions (or angles), the standardization (systematic), objectivity and comprehensiveness of the quality evaluation of the code data set are guaranteed, and the applicability is improved. According to the characterization quality index and content quality index, the defects of the code data set are determined so as to accurately identify the quality defects, thereby facilitating targeted repair; by performing automated or semi-automated repair processing on the quality defects, the repair efficiency of the quality defects of the code data set is effectively improved, the labor cost is reduced, and the consistency and accuracy of the quality defect repair are guaranteed. By generating display information about quality defects and comparative information before and after quality defect repair, users can clearly and intuitively understand the specific information of the corresponding quality defects, the effect of quality defect repair, and the improvement of the quality of the code data set, which helps to improve users' confidence and understanding of the repair operation and the repaired code data set.

[0194] Another embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is read and executed by a processor, the quality analysis and defect repair method of the code data set as described above is implemented.

[0195] In other words, a non-volatile computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the following operations:

[0196] Get the code dataset for model training or tuning;

[0197] Determine the representation quality index and content quality index of the code dataset, wherein the representation quality index is used to refer to the quality of the data presentation form of the code dataset, and the content quality index is used to refer to the quality of the data presentation content of the code dataset;

[0198] Determine the quality defects of the code dataset based on representation quality indicators and content quality indicators;

[0199] Automatic or semi-automatic repair of quality defects;

[0200] Generates display information about quality defects and comparative information before and after the quality defects are repaired.

[0201] The technical solution of the embodiment of the present invention, in essence or in other words, the part that contributes to the prior art, or the whole or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for making a computer device (which can be a personal computer, server, or network device, etc.) or a processor perform all or part of the steps of the method of the embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0202] By storing the computer program corresponding to the quality analysis and defect repair method of the code data set through a computer-readable storage medium, the stability of the computer program corresponding to the quality analysis and defect repair method of the code data set can be ensured when it is read and run by the corresponding processor. By executing the quality analysis and defect repair method of the code data set in this way, the quality analysis and defect repair of the code data set used for training or fine-tuning a large language model can be realized, and it can be used to provide a high-quality code data set for training or fine-tuning a large language model. Specifically, by determining the representation quality index and content quality index of the code data set used for training or fine-tuning a large language model, the quality of the code data set is evaluated from multiple dimensions (or angles), the standardization (systematicity), objectivity and comprehensiveness of the quality evaluation of the code data set are guaranteed, and the applicability is improved. According to the representation quality index and content quality index, the defects of the code data set are determined so as to accurately identify the quality defects, thereby facilitating targeted repair; by performing automated or semi-automated repair processing on the quality defects, the repair efficiency of the quality defects of the code data set can be effectively improved, the labor cost can be reduced, and the consistency and accuracy of the quality defect repair can be guaranteed. By generating display information about quality defects and comparative information before and after quality defect repair, users can clearly and intuitively understand the specific information of the corresponding quality defects, the effect of quality defect repair, and the improvement of the quality of the code data set, which helps to improve users' confidence and understanding of the repair operation and the repaired code data set.

[0203] Although the present invention is disclosed as above, the protection scope of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the protection scope of the present invention.

Claims

1. A method for quality analysis and defect repair of a code data set, characterized in that: include: Get the code dataset for model training or tuning; Determine a representation quality index and a content quality index of the code data set, wherein the representation quality index is used to refer to the quality of the data presentation form of the code data set, and the content quality index is used to refer to the quality of the data presentation content of the code data set; Determining quality defects of the code data set according to the representation quality indicator and the content quality indicator; Performing automated or semi-automated repair processing on the quality defects; Display information about the quality defect and comparison information before and after repair of the quality defect is generated.

2. The method for quality analysis and defect repair of a code data set according to claim 1, characterized in that: The characterization quality index includes at least one of a repetition rate, a format standard rate, a sample defect rate, a code syntax correctness rate, and a code block standard rate.

3. The method for quality analysis and defect repair of a code data set according to claim 2, characterized in that: The automatic or semi-automatic repair process for the quality defect includes: When the quality defect with respect to the repetition rate exists, deleting repeated samples in the code data set; When the quality defect with respect to the format standard rate exists, a regular expression is used to perform repair processing on the corresponding defective sample; When the quality defect regarding the sample defect rate exists, deleting the defective samples in the code data set; When the quality defect regarding the code syntax correctness exists, a second preset model is used to repair the corresponding defective sample; When the quality defect with respect to the standard rate of the code block exists, a third preset model is used to perform repair processing on the corresponding defective sample.

4. The method for quality analysis and defect repair of a code data set according to any one of claims 1 to 3, characterized in that: The content quality indicator includes at least one of the scores of the code dataset regarding intent completeness, response correctness, code logic correctness, and question-answer correspondence.

5. The method for quality analysis and defect repair of a code data set according to claim 4, characterized in that: Determining the representation quality index and content quality index of the code data set includes: Using a first preset expert model to adjust the code data set with respect to the intent completeness, to obtain a first code adjustment data set; inputting the code data set and the first code adjustment data set into a preset universal scoring model, and outputting a first score of the code data set with respect to the intent completeness; determining the score of the code data set with respect to the intent completeness according to the first score obtained at least once; and / or, using a second preset expert model to adjust the code dataset with respect to the correctness of the response to obtain a second code adjustment dataset; inputting the code dataset and the second code adjustment dataset into the preset universal scoring model, and outputting a second score of the code dataset with respect to the correctness of the response; determining the score of the code dataset with respect to the correctness of the response based on the second score obtained at least once; and / or, using a third preset expert model to adjust the code data set with respect to the correctness of the code logic to obtain a third code adjustment data set; inputting the code data set and the third code adjustment data set into the preset universal scoring model, and outputting a third score of the code data set with respect to the correctness of the code logic; determining the score of the code data set with respect to the correctness of the code logic according to the third score obtained at least once; And / or, using a fourth preset expert model to adjust the code dataset with respect to the question-answer correspondence to obtain a fourth code adjusted dataset; inputting the code dataset and the fourth code adjusted dataset into the preset universal scoring model, and outputting a fourth score of the code dataset with respect to the question-answer correspondence; determining the score of the code dataset with respect to the question-answer correspondence based on the fourth score obtained at least once.

6. The method for quality analysis and defect repair of a code data set according to claim 5, characterized in that: Determining the score of the code dataset regarding the intent completeness according to the first score obtained at least once includes: Determining a score of the code dataset regarding the intent completeness based on a first variance and a first average of all the first scores; And / or, determining the score of the code dataset regarding the correctness of the response according to the second score obtained at least once comprises: determining a score of the code data set regarding the correctness of the response based on a second variance and a second mean of all the second scores; And / or, determining the score of the code data set regarding the logical correctness of the code according to the third score obtained at least once comprises: Determining a score of the code data set regarding logical correctness of the code according to a third difference and a third average of all the third scores; And / or, determining the score of the code dataset regarding the question-answer correspondence based on the fourth score obtained at least once includes: The score of the code dataset with respect to the question-answer correspondence is determined based on the fourth variance and the fourth average of all the fourth scores.

7. The method for quality analysis and defect repair of a code data set according to claim 6, characterized in that: Determining the score of the code dataset regarding the intent completeness according to the first variance and the first average of all the first scores comprises: When the first variance is less than or equal to a first preset threshold, the first average value is used as the score of the code data set with respect to the intent completeness; when the first variance is greater than a first preset threshold, a prompt message for performing a manual score with respect to the intent completeness is generated, and in response to receiving a first manual score with respect to the intent completeness, the first manual score is used as the score of the code data set with respect to the intent completeness; And / or, determining the score of the code dataset regarding the correctness of the response according to the second variance and the second average of all the second scores comprises: When the second variance is less than or equal to a second preset threshold, the second average value is used as the score of the code data set regarding the correctness of the response; when the second variance is greater than the second preset threshold, a prompt message for manual scoring of the correctness of the response is generated, and in response to receiving a second manual score regarding the correctness of the response, the second manual score is used as the score of the code data set regarding the correctness of the response; And / or, determining the score of the code data set regarding the logical correctness of the code according to the third party differences and the third average of all the third scores includes: When the third difference is less than or equal to a third preset threshold, the third average value is used as the score of the code data set regarding the correctness of the code logic; when the third difference is greater than the third preset threshold, a prompt message for manual scoring of the correctness of the code logic is generated, and in response to receiving a third manual score regarding the correctness of the code logic, the third manual score is used as the score of the code data set regarding the correctness of the code logic; And / or, determining the score of the code dataset regarding the question-answer correspondence according to the fourth variance and the fourth mean of all the fourth scores includes: When the fourth variance is less than or equal to a fourth preset threshold, the fourth average value is used as the score of the code dataset regarding the correctness of the response; when the fourth variance is greater than the fourth preset threshold, a prompt message is generated regarding performing a manual score on the question-answer correspondence, and in response to receiving a fourth manual score on the question-answer correspondence, the fourth manual score is used as the score of the code dataset regarding the question-answer correspondence.

8. The method for quality analysis and defect repair of a code data set according to claim 4, characterized in that: The automatic or semi-automatic repair process for the quality defect includes: Step 410: When the code data set has a content quality defect regarding at least one of the intent completeness, the response correctness, the code logic correctness, and the question-answer correspondence, determine a first defect description and repair suggestion information regarding the content quality defect through a fifth preset model; Step 420: According to the first defect description and repair suggestion information, a sixth preset model is used to repair the code data set to obtain a corrected code data set; Step 430: Use the first preset model to score the correction code data set to obtain a second content quality score corresponding to the correction code data set, wherein the first preset model includes a first preset expert model, a second preset expert model, a third preset expert model, a fourth preset expert model and a preset general scoring model; Step 440: If the second content quality score is higher than the first content quality score corresponding to the code data set and higher than the fifth preset threshold, determine that the corresponding content quality defect is repaired; if the second content quality score is lower than at least one of the first content quality score and the fifth preset threshold, use the corrected code data set as the code data set in step 410, and repeat steps 410 to 440 until the second content quality score corresponding to the corrected code data set is higher than the first content quality score and higher than the fifth preset threshold; wherein the first content quality score includes at least one of the scores of the code data set regarding intent completeness, response correctness, code logic correctness, and question-answer correspondence.

9. The method for quality analysis and defect repair of a code data set according to any one of claims 1 to 3, characterized in that: After obtaining the code data set for model training or adjustment and before automatically or semi-automatically repairing the quality defects, the code data set quality analysis and defect repair method further includes: Using a preset statistical algorithm to obtain statistical features of the code data set; The automatic or semi-automatic repair process for the quality defect includes: Display information about the statistical feature is generated.

10. A quality analysis and defect repair device for a code data set, characterized in that: include: An acquisition unit, used to acquire a code dataset for model training or adjustment; a quality indicator determination unit, configured to determine a characterization quality indicator and a content quality indicator of the code data set, wherein the characterization quality indicator is used to refer to the quality of the data presentation form of the code data set, and the content quality indicator is used to refer to the quality of the data presentation content of the code data set; A quality defect determination unit, configured to determine the quality defect of the code data set according to the characterization quality indicator and the content quality indicator; A quality defect repair unit, used for automatically or semi-automatically repairing the quality defect; A display unit is used to generate display information about the quality defect and comparison information before and after the quality defect is repaired.

11. An electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the quality analysis and defect repair method of the code data set according to any one of claims 1 to 9 when executing the computer program.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is read and executed by a processor, the method for quality analysis and defect repair of a code data set according to any one of claims 1 to 9 is implemented.

Images