Data processing and code processing method, device, all-in-one machine and storage medium

Abstract

The application provides a data processing and code processing method, device, all-in-one machine and storage medium, wherein the method comprises: obtaining an initial code and a requirement description text corresponding to the initial code; determining a matching result between the initial code and the requirement description text based on a matching query statement carrying the initial code and the requirement description text; in the case that the matching result is matching, generating a test case based on a test verification statement carrying the initial code and the requirement description text, and obtaining a test verification result obtained by running the test case; in the case that the test verification result is passed, taking the initial code and the requirement description text as a training data pair. The method, device, all-in-one machine and storage medium provided by the application can screen high-quality training data pairs, thereby providing high-quality data for supervised training of a model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data processing and code processing method and device, an all-in-one machine and a storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence and natural language processing, large language models have shown strong capabilities on various tasks, such as chat dialogue, text generation, logical reasoning, graph understanding, and programming capabilities.

[0003] Large language models usually need to be trained using large-scale training data to achieve better performance and generalization ability. However, the acquisition and screening of large-scale training data has become an important challenge. Although there is a large amount of text data on the Internet, the quality is uneven, containing errors and inaccurate information. If low-quality data is used for training, the model will learn incorrect knowledge, thereby reducing the performance and reliability of the model. SUMMARY

[0004] The present application provides a data processing and code processing method, device, all-in-one machine and storage medium to solve the problem of uneven quality of training data in the prior art, and to achieve high-quality training data.

[0005] The present application provides a data processing method, comprising:

[0006] obtaining an initial code and a requirement description text corresponding to the initial code;

[0007] determining a matching result between the initial code and the requirement description text based on a matching query statement carrying the initial code and the requirement description text;

[0008] in the case of matching, generating a test case based on a test verification statement carrying the initial code and the requirement description text, and obtaining a test verification result obtained by running the test case;

[0009] in the case of passing, the initial code and the requirement description text are used as a training data pair.

[0010] According to the data processing method provided by the present application, the test verification statement carrying the initial code and the requirement description text is generated, and the test verification result obtained by running the test case is obtained, comprising:

[0011] filling the initial code and the requirement description text into a test verification template to obtain the test verification statement;

[0012] inputting the test verification statement into a test code generation model to obtain the test case and running code for running the test case output by the test code generation model;

[0013] combining and running the initial code, the test case and the running code to obtain a running result, and parsing the running result to obtain a test verification result.

[0014] According to the data processing method provided by the application, the matching result between the initial code and the requirement description text is determined based on the matching query statement carrying the initial code and the requirement description text, and the matching result comprises:

[0015] The initial code and the requirement description text are filled into a preset matching template to obtain the matching query statement;

[0016] The matching query statement is input into a matching model to obtain the matching result output by the matching model.

[0017] According to the data processing method provided by the application, the initial code is obtained, and the initial code comprises:

[0018] The original code and the requirement description text corresponding to the original code are obtained;

[0019] The original code and the requirement description text are subjected to length screening, and / or the original code is subjected to compilation screening;

[0020] The original code that passes the screening is used as the initial code.

[0021] According to the data processing method provided by the application, the original code and the requirement description text are subjected to length screening, and / or the original code is subjected to compilation screening, and the length screening of the original code is based on a preset code length threshold, and the length screening of the requirement description text is based on a preset requirement length threshold.

[0022] The original code is subjected to length screening based on a preset code length threshold, and the requirement description text is subjected to length screening based on a preset requirement length threshold.

[0023] In the case that the original code and the requirement description text corresponding to the original code both pass the length screening, the original code is subjected to compilation, and the original code is subjected to screening based on the compilation result.

[0024] The application further provides a code processing method, comprising:

[0025] obtaining a target requirement text;

[0026] Based on the code model, the target requirement text is applied for code processing, and a target processing result corresponding to the target requirement text is obtained.

[0027] The code model is obtained based on training data pairs, and the training data pairs are obtained based on any of the data processing methods.

[0028] The application further provides a data processing apparatus, comprising:

[0029] A data acquisition unit is configured to acquire an initial code and a requirement description text corresponding to the initial code.

[0030] A matching and screening unit is configured to determine a matching result between the initial code and the requirement description text based on a matching query statement carrying the initial code and the requirement description text.

[0031] A test verification unit is configured to, when the matching result is matching, generate a test case and acquire a test verification result obtained by running the test case based on a test verification statement carrying the initial code and the requirement description text.

[0032] A data determination unit is configured to, when the test verification result is passed, take the initial code and the requirement description text as a training data pair.

[0033] The application further provides a code processing apparatus, comprising:

[0034] A text acquisition unit is configured to acquire a target requirement text.

[0035] A code processing unit is configured to, based on a code model, apply the target requirement text for code processing, and obtain a target processing result corresponding to the target requirement text.

[0036] The code model is obtained based on training data pairs, and the training data pairs are obtained based on any of the data processing methods.

[0037] The application further provides a code processing all-in-one machine, comprising a memory, a processor, a code model stored in the memory, and a computer program stored in the memory and executable on the processor.

[0038] When the processor executes the computer program, the code model is invoked to implement the code processing method.

[0039] The application further provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement any of the data processing methods or code processing methods.

[0040] The application further provides a computer program product comprising a computer program and / or a code model, the computer program being executed by a processor to invoke the code model to implement the code processing method as described above.

[0041] The application provides a data processing and code processing method, device, all-in-one machine and storage medium, which performs matching screening on the initial code and the requirement description text based on a matching query statement, and further performs test verification on the running result of the initial code based on a test verification statement in the case of matching between the initial code and the requirement description text, so as to obtain high-quality training data pairs, thereby providing high-quality data for supervised training of the model, enabling the model to learn accurate and consistent knowledge, and improving the performance and reliability of the model. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0043] Figure 1 is one of the flowcharts of the data processing method provided by the application;

[0044] Figure 2 is the flowchart of step 130 in the data processing method provided by the application;

[0045] Figure 3 is the flowchart of step 120 in the data processing method provided by the application;

[0046] Figure 4 is the second flowchart of the data processing method provided by the application;

[0047] Figure 5 is the flowchart of the code processing method provided by the application;

[0048] Figure 6 is the structural schematic diagram of the data processing device provided by the application;

[0049] Figure 7 is the structural schematic diagram of the code processing device provided by the application;

[0050] Figure 8 is the structural schematic diagram of the all-in-one machine provided by the application. DETAILED DESCRIPTION

[0051] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in conjunction with the drawings in the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0052] With the rapid development of artificial intelligence and natural language processing, large language models have shown strong capabilities on various tasks, such as chat dialogue, text generation, logical reasoning, graph understanding, and programming capabilities. Large language models usually need to be trained using large-scale training data to obtain better performance and generalization ability. However, the acquisition and screening of large-scale training data has become an important challenge.

[0053] For example, for the training of model programming capabilities, although there are some high-quality datasets on existing programming websites that can be directly used as supervised training data for models, such as CodeContests and CodeNet datasets, the amount of such data is generally small and is far from enough as training data. Therefore, some professional institutions will screen the code data on GitHub (a software project hosting platform) through certain rules and finally organize them into supervised datasets, such as CodeSearchNet and TheVault datasets, but the processing flow of such data is too single, only using rule-based methods for screening, which easily leads to uneven quality of data in the obtained dataset. If the data in such dataset is directly used as training data, it will cause the model to learn incorrect knowledge, thereby reducing the performance and reliability of the model. To this end, the embodiments of the present application provide a data processing method to overcome the above defects.

[0054] Figure 1 is one of the flowcharts of the data processing method provided by the present application, as shown in Figure 1 The method comprises the following steps:

[0055] Step 110, obtaining an initial code and a requirement description text corresponding to the initial code;

[0056] Specifically, the initial code can be code examples in different programming tasks and scenarios directly obtained from various open source datasets, or code obtained by preprocessing the original code obtained from various open source datasets, and the embodiments of the present application do not make specific limitations thereto. Here, the open source dataset can be CodeSearchNet dataset, TheVault dataset, etc., and the initial code can cover various programming languages (such as Python, Java, C++, etc.) and different programming tasks (such as data processing, algorithm implementation, graphical interface design, etc.).

[0057] The above requirement description text refers to the natural language description of the requirements or functional requirements corresponding to the initial code, and the requirement description text can be used to explain the function, purpose and design requirements of the initial code. It can be understood that the open source dataset usually includes a plurality of data pairs, each data pair includes a requirement description text and a code segment corresponding to the requirement description text, and after downloading various open source datasets from the network, the initial code and the requirement description text corresponding to the initial code can be obtained.

[0058] Step 120, determining the matching result between the initial code and the requirement description text based on the matching query statement carrying the initial code and the requirement description text;

[0059] It should be noted that the data pairs in the open source dataset are extracted based on the annotations and codes in various open source source code websites (such as GitHub, Gitee, etc.). Considering that some annotations cannot well explain the meaning of the code, resulting in a low matching degree between the obtained requirement description text and the initial code, if such data pairs are directly used as training data pairs, the knowledge learned by the model will not be accurate enough, thereby affecting the performance of the model. Therefore, the embodiments of the present application judge the matching degree between the requirement description text and the initial code through the matching query statement, so as to screen the data pairs matched with the requirement description text and the initial code.

[0060] Specifically, the matching query statement refers to the query statement carrying the initial code and the requirement description text, which can be used to find data matching the specified conditions and rules. After obtaining the initial code and the requirement description text, the specified conditions and rules can be formed by the initial code and the requirement description text, so as to obtain the matching query statement.

[0061] For example, a matching query template "Please judge whether the code [code] implements the function described below [prompt]" can be set in advance, and after obtaining the initial code and the requirement description text, the specific content of the initial code can be used to replace the code field in the template, and the specific content of the requirement description text can be used to replace the prompt field in the template, so as to obtain the complete matching query statement.

[0062] In an embodiment, the obtained matching query statement can be input into a code interpretation model, and the meaning and logic of the initial code are understood through the code interpretation model to obtain an interpretation and explanation corresponding to the initial code, and the interpretation and explanation are matched with the input requirement description text in terms of similarity, so as to obtain a matching result between the initial code and the requirement description text, that is, in the case where the similarity is greater than or equal to a preset similarity threshold, it is indicated that the initial code can implement the function corresponding to the requirement description text, and at this time, it can be determined that the matching result is matching; in the case where the similarity is less than the preset similarity threshold, it is indicated that the initial code cannot completely implement the function corresponding to the requirement description text, and therefore it can be determined that the matching result is not matching.

[0063] It can be understood that the code interpretation model refers to a machine learning model, which can extract the information of the code by learning the structure, semantics and context of a large amount of code, so as to generate an explanatory description related to the code.

[0064] Step 130, in the case where the matching result is matching, based on the test verification statement carrying the initial code and the requirement description text, a test case is generated and a test verification result obtained by running the test case is acquired;

[0065] It should be noted that for the training of the model programming ability, the correctness of the code running result is very important, which will directly affect the effect of the trained model. Considering that the running result of some initial codes in the open source data set is not necessarily correct, therefore, in the case where the initial code and the requirement text are matched with each other, the initial code and the requirement description text are further tested and verified in the embodiment of the application, so as to filter out training data pairs with correct running results, thereby further improving the quality of the training data pairs.

[0066] Specifically, after the matching initial code and the requirement description text are filtered out, the corresponding test verification statement can be obtained according to the requirement description text and the initial code, and the corresponding test case is generated according to the test verification statement. Here, the input conditions required to be provided can be determined according to the test verification statement, for example, the input conditions can be the parameters of the function, the attributes of the object, etc., and then the corresponding test case is generated according to the test verification statement and the given input conditions. The test case can be generated by using methods such as equivalence class division and boundary value analysis, which can cover different input situations and boundary conditions for more comprehensive test verification.

[0067] After the test case is generated, the generated test case can be run through scripting programming or calling an automated testing tool, and a test verification result is obtained. It can be understood that the test case can include various possible input conditions and boundary conditions, and an expected output result for a given input. For each test case, the actual output result can be compared with the expected output result. If the two are consistent, the test verification passes; if the two are inconsistent, the test verification fails. By generating and running the test case, it can be determined whether the initial code works as expected and whether it meets the given requirements, thereby obtaining qualified supervised training data pairs.

[0068] Step 140, in the case where the test verification result is passed, the initial code and the requirement description text are taken as a training data pair.

[0069] Specifically, the test verification result is passed, indicating that the initial code can achieve the expected function. In this case, the initial code and the requirement description text can be taken as a training data pair. By performing the above steps, a large number of high-quality training data pairs can be screened, thereby providing high-quality data for supervised training of the code model.

[0070] Here, the training data pair refers to a set of data used for model training in machine learning or deep learning, wherein each data sample is composed of an input and a corresponding output. The training data pair is usually used for a supervised learning task, in which the requirement description text can be taken as the input and the initial code as the target output, or the initial code can be taken as the input and the requirement description text as the target output. The model learns the mapping relationship between the input and the output to obtain the code capability. By screening high-quality training data pairs, the performance and accuracy of the model code processing can be improved.

[0071] The data processing method provided by the embodiment of the present application can match and screen the initial code and the requirement description text based on the matching query statement. In the case where the initial code matches the requirement description text, the running result of the initial code is further tested and verified based on the test verification statement, so as to obtain a high-quality training data pair, thereby providing high-quality data for supervised training of the model, enabling the model to learn accurate and consistent knowledge, and improving the performance and reliability of the model.

[0072] Based on the above embodiment, Figure 2 is a flowchart of step 130 in the data processing method provided by the present application, as Figure 2 shown, step 130 specifically includes:

[0073] Step 131, filling the initial code and the requirement description text into a test verification template to obtain a test verification statement;

[0074] Specifically, the test verification template is a predefined template for matching the generated test verification statement, which can use specific tags to represent variables in the template. After screening the matched initial code and requirement description text, it can be filled into the test verification template, that is, the initial code and the requirement description text are replaced with the corresponding variables in the test verification template, so as to obtain the test verification statement.

[0075] For example, the test verification template can be:

[0076] {“query”:“you are a senior code developer, please generate [num] unit test cases and executable code according to the following requirement description [prompt] and code implementation [code]”,“answer”:“test case and running code”}

[0077] Wherein, num represents the number of test cases generated by pre-setting, for example, num can be 2, which can also be set according to actual needs. After obtaining the matched initial code and requirement description text, the specific content of the initial code can be used to replace the code field in the template, and the requirement description text can be used to replace the prompt field in the template, so as to obtain the test verification statement.

[0078] Step 132, input the test verification statement into the test code generation model to obtain the test case and the running code for running the test case output by the test code generation model;

[0079] Specifically, after obtaining the test verification statement, it can be input into the test code generation model, so as to obtain the output test case and running code. Here, the running code refers to the code for running the test case, which includes the code for calling the initial code.

[0080] The above test code generation model is a model based on machine learning or deep learning, which aims to automatically generate test cases for the initial code. Before step 132 is executed, the test code generation model can also be pre-trained. Specifically, it can be trained by the following way: collecting a large amount of test code dataset for training model, which can include existing test cases, test scripts, code comments, etc., selecting a suitable initial model, training the initial model based on the prepared test code dataset, so as to obtain the test code generation model.

[0081] For example, the test code generation model can be a trained large language model Expert Model. By inputting the test verification statement into the test code generation model, it can automatically generate test cases and corresponding running code according to the given initial code and requirement description text and output.

[0082] Step 133, combining the initial code, test case and running code and running to obtain a running result, and parsing the running result to obtain a test verification result.

[0083] Specifically, after obtaining the test case and the running code, the initial code, the test case and the running code can be combined into a code file according to a rule. For example, a code file structure can be defined in advance, and the initial code, the test case and the running code can be placed in different code blocks or code segments. At the beginning of the code file, the required libraries and modules are imported. The combined code file contains the initial code, the code for calling the initial code by the unit test, and the dependencies required for code running (such as header files, various libraries and modules, etc.).

[0084] At the end of the code file, code can be added to run the test. For example, it can be a function call or a specific command. When running the test, the test framework or tool will automatically execute the test case and output the corresponding running result. After obtaining the running result, the running result can be parsed. For example, key information in the running result can be extracted through string processing, regular expressions, JSON parsing and other methods, so as to obtain the test verification result.

[0085] It can be understood that in the unit test, additional test cases can be added through redundant design to cover more cases and boundary conditions, so as to ensure the correctness and stability of the code under various conditions. Therefore, when the test case is generated by the test code generation model, two or more test cases can be generated. These test cases are independent of each other. If at least one test case passes, it is considered that the initial code passes the unit test. At this time, it can be determined that the test verification result is passed, and the initial code and the requirement description text that pass the unit test are retained, and the initial code and the requirement description text that do not pass the unit test are filtered.

[0086] The method provided by the embodiment of the application can improve the efficiency of testing and verifying the initial code and the requirement description text, and further improve the efficiency of data processing.

[0087] Based on any of the above embodiments, Figure 3 is a flowchart of step 120 in the data processing method provided by the application, as shown in Figure 3 Step 120 specifically includes:

[0088] Step 121, filling the initial code and the requirement description text into a preset matching template to obtain a matching query statement;

[0089] Step 122, input the matching query statement into the matching model to obtain a matching result output by the matching model.

[0090] Specifically, the preset matching template is a predefined template for matching specific information, which can use specific labels to represent variables in the template. After obtaining the initial code and the requirement description text, the initial code and the requirement description text can be filled into the preset matching template, that is, the initial code and the requirement description text are respectively replaced with corresponding variables in the preset matching template, so as to obtain the matching query statement. Then, the matching query statement is input into the matching model as an input of the matching model, the matching query statement is analyzed by the matching model, and the matching result between the initial code and the requirement description text is output in a given format.

[0091] Exemplarily, the matching model can be a trained large language model, for example, an expert model (ExpertModel). Correspondingly, the preset matching template can be:

[0092] {“query”:“you are a code review expert, please judge whether the code [code] implements the following described function [prompt]”,“answer”:“yes / no”}

[0093] After obtaining the initial code and the requirement description text, the specific content of the initial code can be used to replace the code field in the template, and the requirement description text can be used to replace the prompt field in the template, so as to obtain the matching query statement. The matching query statement is input into the expert model, and the output matching result “yes” or “no” can be obtained. According to the matching result, the data pairs are filtered, the data pairs matching the prompt and the code are retained, and the data pairs not matching the prompt and the code are filtered.

[0094] The method provided by the embodiment of the application can quickly obtain the matching result output by the matching model by inputting the matching query statement into the matching model, thereby improving the efficiency of matching and filtering the initial code and the requirement description text, and ensuring the accuracy of the matching result while improving the efficiency of data processing.

[0095] Based on any of the above embodiments, in step 110, the initial code is obtained, including:

[0096] Step 111, obtaining the original code and the requirement description text corresponding to the original code.

[0097] Step 112, length filtering the original code and the requirement description text, and / or compiling the original code.

[0098] Step 113, taking the original code that passes the filtering as the initial code.

[0099] It should be noted that considering that the lengths of the demand description texts and the initial codes in the open source data sets downloaded from the network are different, some of which even exceed the maximum length of the model running, and some of which are too short to clearly describe the demand, in addition, the syntax, format, function call and the like of some initial codes may be wrong, resulting in the code unable to run normally. In this regard, in order to obtain a high-quality training data pair, the embodiment of the application performs length screening and / or compilation screening on the original code to preliminarily screen the initial code with a qualified length and capable of running normally.

[0100] Specifically, the original code refers to the code examples in different programming tasks and scenarios directly obtained from various open source data sets, and the demand description text refers to the textual description or explanation corresponding to the original code obtained from various open source data sets. After obtaining the original code and the demand description text, the original code and the demand description text can be screened in length, and in the case that the lengths of the original code and the demand description text are qualified, the original code is further automatically compiled to check the syntax, format, function call and the like of the initial code, so as to screen the original code with a qualified length and capable of running normally.

[0101] Here, when the original code is screened in compilation, appropriate tools and methods can be selected according to the programming language and development environment of the original code to perform automatic compilation, for example, for compiled languages (such as C, C++, etc.), a build tool (such as Make, CMake) can be used to automatically compile according to a predefined configuration file or script; for script languages (such as Python, JavaScript, etc.), a script management tool (such as pip, Babel, etc.) can be used to automatically compile.

[0102] By automatically compiling the original code, the syntax, format, function call and the like of the original code can be checked to ensure that the original code can run normally. After the compilation screening of all original codes is completed, the original code that passes the compilation can be retained, and the demand description text corresponding to the original code is the demand description text corresponding to the initial code.

[0103] Based on the above embodiment, step 112 specifically includes:

[0104] Screening the original code in length based on a preset code length threshold, and screening the demand description text in length based on a preset demand length threshold;

[0105] In the case that the original code and the demand description text corresponding to the original code both pass the length screening, the original code is compiled, and the original code is screened based on the compilation result.

[0106] Specifically, for the code of the compiled language, the function corresponding to the requirement description text cannot be well implemented when the code length is too short, therefore, in order to control the quality of the code, the original code can be filtered based on the preset code length threshold. Here, the preset code length threshold can be a preset code line number, which can be set according to actual needs, for example, the preset code length threshold can be 5, in the case that the number of lines of the original code is less than 5, the original code can be filtered out to ensure the quality of the initial code obtained, thereby improving the quality of the training data pair obtained after the final screening. It should be understood that for the code of the script language, even if the length is short, it can well implement the function corresponding to the requirement description text, therefore, the original code of the script language is not subjected to length filtering.

[0107] Considering that the model input has a tokens limit, i.e., the maximum number of words or tokens of the input text that the model can process is limited, therefore, the requirement description text can be filtered based on the preset requirement length threshold. Here, the preset requirement length threshold refers to a predefined string length threshold contained in the requirement description text, which can be set according to the maximum input of the model, for example, the preset requirement length threshold can be 1024, in the case that the length of the requirement description text exceeds the preset requirement length threshold, the requirement description text and its corresponding original code can be filtered out to ensure that the length of the initial code and its corresponding requirement description text obtained after the screening meets the requirements of the model input.

[0108] Compared with the compilation screening of the original code, the length screening of the original code and the requirement description text is relatively faster, therefore, in the embodiment of the present application, in the case that the original code and the requirement description text both pass the length screening, the original code is further compiled, which can reduce the number of original codes that need to be compiled, thereby facilitating the improvement of the efficiency of data processing.

[0109] Based on any of the above embodiments, Figure 4 is a flowchart of the data processing method provided by the present application, as shown in Figure 4 , the method comprises:

[0110] S1, obtaining a data pair to be processed;

[0111] Here, a batch of data pairs to be processed can be obtained based on various open source data sets, each data pair comprising a requirement description text and an original code corresponding to the requirement description text.

[0112] S2, length screening of the original code and the requirement description text;

[0113] Considering that the model input has a token limit, data pairs whose requirement description text length exceeds the preset requirement length threshold can be filtered out. In addition, in order to control the code quality, data pairs whose code line number is less than the preset code length threshold can be filtered out, so as to filter out data pairs whose original code and requirement description text both pass the length screening.

[0114] S3, in the case of passing the screening in step S2, automatically compiling the original code;

[0115] Here, whether it is a compiled language or a script language, it needs to be compiled before it can be run. The difference is that the compiled language needs to be run after explicit compilation, while the script language is automatically compiled and run. By compiling the original code, syntax, format, function call errors and other errors in the original code can be checked, and preliminary screening of the data pair is completed. The data pair that passes the compilation is retained, and the original code that passes the compilation is used as the initial code. The data pair that does not pass the compilation is filtered out.

[0116] S4, in the case of passing the screening in step S3, based on the matching query statement carrying the initial code and the requirement description text, determining the matching result between the requirement description text and the initial code;

[0117] Here, after obtaining the initial code and the requirement description text through step S3, the initial code and the requirement description text can be filled into a preset matching template to obtain a matching query statement. Then, the matching query statement is input into a matching model to obtain a matching result output by the matching model.

[0118] It can be understood that the matching model can be a trained large language model, such as an expert model (ExpertModel). The preset matching template can be {“query”:“you are a code review expert, please judge whether the code [code] implements the function [prompt] described below”,“answer”:“yes / no”}. The query needs to be spliced with the requirement description text and the initial code in each data pair to obtain a matching query statement. Then, the matching query statement is input into the matching model, and the matching result is output according to the matching result to screen the data pair. The data pair whose requirement description text and initial code match is retained, and the data pair whose requirement description text and initial code do not match is filtered out.

[0119] S5, in the case of passing the screening in step S4, generating corresponding test cases and test code according to the requirement description text and the initial code and running;

[0120] Here, the data pair screened through step S4 also needs to verify the correctness of the code running result. A test verification template can be set in advance, such as {“query”:“You are a senior code developer. Please describe [prompt] and implement [code] according to the following requirements to generate [num] unit test cases and executable code”,“answer”:“test case and running code”}. The initial code and requirement description text screened through step S4 are filled into the test verification template to obtain a test verification statement.

[0121] Subsequently, the test verification statement is input into the test code generation model to obtain output test cases and running code for running the test cases. According to the rules, the initial code, test cases, and running code are combined into a code file, the code file is run to obtain a running result, and the running result is parsed. For two test cases, only one of them needs to pass the test verification, and the data pair that passes the test verification is retained, and the data pair that does not pass the test verification is filtered.

[0122] After the above several steps of screening, the initial code and requirement description text that pass the screening can be used as training data pairs, so as to ensure that the training data pairs obtained by screening have high quality and can provide reliable supervised training data for the model.

[0123] Based on any of the above embodiments, Figure 5 is a flowchart of the code processing method provided by the present application, as shown in Figure 5 , the method comprises:

[0124] Step 510, obtaining a target requirement text;

[0125] Step 520, based on the code model, applying the target requirement text for code processing to obtain a target processing result corresponding to the target requirement text; the code model is obtained based on the training data pair, and the training data pair is obtained based on the data processing method as described in any of the above embodiments.

[0126] Specifically, after the high-quality training data pair is screened according to the data processing method in the above embodiments, the training data pair can be used as the training data of the model, and the code model is trained accordingly. Therefore, the target requirement text can be applied to the code processing based on the code model to obtain the corresponding target processing result, so as to provide a more intelligent and efficient programming auxiliary tool and development environment for the developer and improve the development efficiency.

[0127] Here, the target requirement text can be a text directly input by the user, or a text obtained by recognizing voice data input by the user, and the embodiments of the present application do not make specific limitations.

[0128] In an embodiment, the target requirement text can be a textual description of a user for a certain programming task or requirement. After obtaining the target requirement text, it can be input into the code model, and code generation is automatically performed based on the code model to obtain the target code corresponding to the target requirement text. The target code is the target processing result, which can help developers create and implement code more efficiently and improve development efficiency.

[0129] In another embodiment, the target requirement text can be a textual description including requirement description text and partial code segments. After obtaining the target requirement text, it is input into the code model. The code model automatically completes and continues writing the current code based on the current input code context, requirement description, and language specifications to obtain a complete code segment. The complete code segment is the target processing result, which can improve the coding speed and accuracy of developers and reduce input errors and redundancy.

[0130] In yet another embodiment, the target requirement text can be a textual description carrying program code. After obtaining the target requirement text, it is input into the code model. The code model interprets the content of the program code to obtain the corresponding code interpretation. The code interpretation is the target processing result, which can help developers understand and debug complex code segments.

[0131] It can be understood that the code model is obtained based on training data pairs. Each training data pair includes sample code and sample requirement description text. The sample code can provide effective programming examples for the model, enabling it to learn the syntax, structure, logic, and patterns of the code, thereby improving the model's understanding and generation capabilities for programming tasks. The sample requirement description text can provide the context and guidance of the task for the model, helping the model to understand and accurately generate the corresponding code or complete and continue writing the code.

[0132] By using high-quality training data pairs for training, the code model can understand the requirements of different code processing tasks and generate high-quality code or corresponding explanations that meet the requirements, which helps to improve the performance and accuracy of the code model, thereby providing better programming support for developers.

[0133] The code processing method provided by the embodiments of the present application can obtain the target requirement text input by the user through the code model and automatically implement code processing to obtain the corresponding target processing result, thereby providing developers with more intelligent and efficient programming auxiliary tools and development environments, reducing the workload of writing repetitive code, accelerating problem solving and development speed, and reducing the risk of code errors.

[0134] Based on any of the above embodiments, Figure 6is a structural schematic diagram of the data processing apparatus provided by the present application, as shown in the figure, the apparatus comprises: Figure 6

[0135] The data acquisition unit 610 is configured to acquire an initial code and a requirement description text corresponding to the initial code.

[0136] The matching screening unit 620 is configured to determine a matching result between the initial code and the requirement description text based on a matching query statement carrying the initial code and the requirement description text.

[0137] The test verification unit 630 is configured to, in a case where the matching result is matching, generate a test case and acquire a test verification result obtained by running the test case based on a test verification statement carrying the initial code and the requirement description text.

[0138] The data determination unit 640 is configured to, in a case where the test verification result is passing, take the initial code and the requirement description text as a training data pair.

[0139] The data processing apparatus provided by the embodiment of the present application matches the initial code and the requirement description text based on the matching query statement, in a case where the initial code matches the requirement description text, further tests and verifies the running result of the initial code based on the test verification statement, so as to obtain a high-quality training data pair, thereby providing high-quality data for supervised training of the model, making the model learn accurate and consistent knowledge, and improving the performance and reliability of the model.

[0140] Based on any one of the above embodiments, the test verification unit 630 is specifically configured to:

[0141] fill the initial code and the requirement description text into a test verification template to obtain a test verification statement;

[0142] input the test verification statement into a test code generation model to obtain a test case output by the test code generation model and a running code used for running the test case;

[0143] combine and run the initial code, the test case and the running code to obtain a running result, analyze the running result to obtain a test verification result.

[0144] Based on any one of the above embodiments, the matching screening unit 620 is specifically configured to:

[0145] fill the initial code and the requirement description text into a preset matching template to obtain a matching query statement;

[0146] input the matching query statement into a matching model to obtain a matching result output by the matching model.

[0147] ​Based on any of the above embodiments, the data acquisition unit 610 specifically comprises:

[0148] An acquisition subunit is configured to acquire the original code and the requirement description text corresponding to the original code.

[0149] A screening subunit is configured to perform length screening on the original code and the requirement description text, and / or perform compilation screening on the original code.

[0150] A determination subunit is configured to determine the original code that passes the screening as the initial code.

[0151] Based on any of the above embodiments, the screening subunit is specifically configured to:

[0152] Perform length screening on the original code based on a preset code length threshold, and perform length screening on the requirement description text based on a preset requirement length threshold.

[0153] In a case where the original code and the requirement description text corresponding to the original code both pass the length screening, perform compilation on the original code, and perform screening based on the compilation result.

[0154] Based on any of the above embodiments, Figure 7 is a structural schematic diagram of a code processing apparatus provided by the present application, as Figure 7 indicated, the apparatus comprises:

[0155] A text acquisition unit 710 is configured to acquire target requirement text.

[0156] A code processing unit 720 is configured to perform code processing based on a code model and the target requirement text to obtain a target processing result corresponding to the target requirement text; the code model is obtained based on training data pairs, and the training data pairs are obtained based on any of the above data processing methods.

[0157] The code processing apparatus provided by the embodiment of the present application can acquire target requirement text input by a user and automatically perform code processing to obtain a corresponding target processing result, thereby providing a more intelligent and efficient programming auxiliary tool and development environment for a developer, reducing the workload of writing repetitive code, accelerating problem solving and development speed, and reducing the risk of code errors.

[0158] Figure 8 An example of an entity structure schematic diagram of a code processing all-in-one machine is shown in Figure 8As shown, the code processing all-in-one machine can include a processor 810, a communications interface 820, a memory 830, and a communications bus 840, wherein the processor 810, the communications interface 820, and the memory 830 complete mutual communication through the communications bus 840, the memory 830 stores a code model and a computer program that can run on the processor 810. The processor 810 can call the code model and the computer program in the memory 830 to execute a code processing method, the method including: obtaining a target requirement text; based on the code model, applying the target requirement text for code processing to obtain a target processing result corresponding to the target requirement text; the code model is obtained based on training data, and the training data pair is obtained based on the above data processing method.

[0159] Here, the code processing all-in-one machine can be an all-in-one machine that privately deploys a code model. The code model and the computer program for calling the code model stored in the memory in the code processing all-in-one machine can be obtained by downloading and installing a computer program installation package or an update package obtained based on a model provider, wherein the code model can be placed in the installation package or the update package in the form of a code file.

[0160] In addition, the computer program in the memory 830 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0161] In another aspect, the present application also provides a computer program product comprising a computer program and / or code model, the computer program being stored on a non-transitory computer-readable storage medium, and the computer program being executable by a processor to cause a computer to perform the data processing method provided by any of the above methods, the method comprising: obtaining an initial code and a requirement description text corresponding to the initial code; determining a matching result between the initial code and the requirement description text based on a matching query statement carrying the initial code and the requirement description text; in a case where the matching result is matching, generating a test case based on a test verification statement carrying the initial code and the requirement description text, and obtaining a test verification result obtained by running the test case; and in a case where the test verification result is passed, taking the initial code and the requirement description text as a training data pair.

[0162] In addition, the computer can also invoke the code model to perform the code processing method provided by any of the above methods, the method comprising: obtaining a target requirement text; performing code processing on the target requirement text based on the code model to obtain a target processing result corresponding to the target requirement text; and the code model being obtained based on a training data pair, the training data pair being obtained based on the data processing method.

[0163] It can be understood that the user can download the computer program product provided by the model provider for installation or update, for example, the above computer program product can be downloaded from a cloud server and installed, so that the above code processing method can be executed on a local terminal to obtain the service provided by the code model.

[0164] In another aspect, the present application also provides a computer program product comprising a computer program and / or code model, the computer program being stored on a non-transitory computer-readable storage medium, and the computer program being executable by a processor to cause a computer to perform the data processing method provided by any of the above methods, the method comprising: obtaining an initial code and a requirement description text corresponding to the initial code; determining a matching result between the initial code and the requirement description text based on a matching query statement carrying the initial code and the requirement description text; in a case where the matching result is matching, generating a test case based on a test verification statement carrying the initial code and the requirement description text, and obtaining a test verification result obtained by running the test case; and in a case where the test verification result is passed, taking the initial code and the requirement description text as a training data pair.

[0165] In addition, the computer can also invoke the code model to perform the code processing method provided by any of the above methods, the method comprising: obtaining a target requirement text; performing code processing on the target requirement text based on the code model to obtain a target processing result corresponding to the target requirement text; and the code model being obtained based on a training data pair, the training data pair being obtained based on the data processing method.

[0166] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0167] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0168] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that, include: Obtain the initial code and the corresponding requirement description text; Based on the matching query statement carrying the initial code and the requirement description text, determine the matching result between the initial code and the requirement description text; If the matching result is a match, test cases are generated based on the test verification statements carrying the initial code and the requirement description text, and the test verification results obtained by running the test cases are obtained. If the test verification result is successful, the initial code and the requirement description text will be used as training data pairs.

2. The data processing method according to claim 1, characterized in that, The process of generating test cases and obtaining test verification results by running the test cases, based on test verification statements carrying the initial code and the requirement description text, includes: Fill the test verification template with the initial code and the requirement description text to obtain the test verification statement; The test verification statement is input into the test code generation model to obtain the test cases and the running code for running the test cases output by the test code generation model. The initial code, the test cases, and the running code are combined and run to obtain the running results. The running results are then parsed to obtain the test verification results.

3. The data processing method according to claim 1, characterized in that, The step of determining the matching result between the initial code and the requirement description text based on a matching query statement carrying the initial code and the requirement description text includes: The initial code and the requirement description text are filled into the preset matching template to obtain the matching query statement; The matching query statement is input into the matching model to obtain the matching result output by the matching model.

4. The data processing method according to any one of claims 1 to 3, characterized in that, The process of obtaining the initial code includes: Obtain the original code, and the corresponding requirement description text; The original code and the requirement description text are filtered by length, and / or the original code is filtered by compilation. The original code that passed the screening will be used as the initial code.

5. The data processing method according to claim 4, characterized in that, The length filtering of the original code and the requirement description text, and / or the compilation filtering of the original code, include: The original code is filtered by length based on a preset code length threshold, and the requirement description text is filtered by length based on a preset requirement length threshold. If both the original code and the corresponding requirement description text are filtered by length, the original code is compiled, and the filtering is performed based on the compilation results.

6. A code processing method, characterized in that, include: Obtain the target requirement text; Based on the code model, the target requirement text is applied to the code processing to obtain the target processing result corresponding to the target requirement text; The code model is obtained based on training data pairs, which are obtained based on the data processing method as described in any one of claims 1 to 5.

7. A data processing apparatus, characterized in that, include: A data acquisition unit is used to acquire initial code and a requirement description text corresponding to the initial code; The matching and filtering unit is used to determine the matching result between the initial code and the requirement description text based on the matching query statement carrying the initial code and the requirement description text; The test verification unit is used to generate test cases and obtain test verification results obtained by running the test cases, based on test verification statements carrying the initial code and the requirement description text, when the matching result is a match. The data determination unit is used to use the initial code and the requirement description text as training data pairs if the test verification result is passed.

8. A code processing all-in-one machine, characterized in that, Includes a memory, a processor, and a code model stored in the memory and a computer program stored in the memory that can run on the processor; When the processor executes the computer program, it invokes the code model to implement the code processing method as described in claim 6.

9. A computer program product, characterized in that, It includes a computer program and / or a code model, wherein the computer program is executed by a processor to invoke the code model to implement the code processing method as described in claim 6.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data processing method as described in any one of claims 1 to 5 or the code processing method as described in claim 6.

Images