Automatic construction method and equipment for fragment-level alignment data and readable storage medium

Abstract

The invention provides an automatic construction method and device for fragment-level alignment data and a readable storage medium, belongs to the technical field of data processing, and aims to solve the problems that program-level alignment data constructed in an existing data alignment method is not beneficial to grammar alignment knowledge learning by a model, and how to construct fragment-level comparison data is not mentioned. Comprising the steps of S1, inserting annotations into a source language program based on a large model; s2, rewriting a target program according to the source language program in which the annotation is inserted, and generating the annotation consistent with the source language program in content and sequence in the target program; and S3, segmenting the source language program and the target program into a plurality of code snippets containing corresponding annotations, and performing one-to-one matching on the code snippets of the source language program and the target program according to the annotations of the code snippets to obtain fragment-level alignment data.

Description

Technical Field

[0001] The present invention relates to a method, a device and a readable storage medium for automatically constructing fragment-level alignment data, and belongs to the technical field of data processing. Background Art

[0002] The code translation task involves translating a piece of code from a source language to a target language. Using large models to automatically construct training data for code translation tasks is a common data augmentation method. From the perspective of data granularity, training data can be divided into program-level alignment and fragment-level alignment. Program-level alignment data typically refers to complete programs, while fragment-level alignment typically refers to small snippets of code, most of which cannot be directly executed. Program-level alignment data helps the model fully learn semantic alignment between programming languages, but due to its length, it is not conducive to the model learning syntactic alignment knowledge. However, fragment-level alignment data, due to its short length, naturally helps the model learn syntactic alignment knowledge. Most current work in the field uses large models to automatically construct program-level alignment data, and no work has studied how to automatically construct fragment-level alignment data. Summary of the Invention

[0003] The present invention aims to solve the problem that the program-level alignment data constructed in the existing data alignment method is not conducive to the model learning grammatical alignment knowledge, and does not mention how to construct fragment-level alignment data. Therefore, a method, device and readable storage medium for automatically constructing fragment-level alignment data are proposed.

[0004] The technical solution adopted by the present invention to solve the above problems is: the method for automatically constructing fragment-level alignment data proposed by the present invention includes: Step 1: Insert annotations into the source language program based on the big model; Step 2: Input the annotated source program and target program into the large model, rewrite the target program based on the annotated source program, and generate annotations in the target program that are consistent with the source program in content and order. Step 3: Split the source language program and the target program into several code snippets containing corresponding comments, and match the code snippets of the source language program and the target program one by one according to the comments of the code snippets to obtain snippet-level alignment data.

[0005] Furthermore, inserting comments into the source language program in step 1 includes: For a given source language program s , using a large model M Source language program s Insert comments into various types of code to generate new source language programs ,in, For thek Notes, For the k code snippets.

[0006] Furthermore, step 2 specifically includes: A set of programming languages ​​for a given target program and the original target program set ,in Indicates the target programming languages, Indicates Original target program written, large language model, M Receiving new source language programs as well as As input, according to the new source language program Rewrite the target program by rewriting the comment content and comment order in the target program, and generate a new source language program in the target program Output the rewritten target program with the same content and order as the comments in , ,in, express No. A code snippet, Indicates the annotations, and then obtain a new set of target programs .

[0007] Furthermore, step 3 specifically includes: New source language program As target program , based on the target program Building a new target program collection ,in , for each pair of target programs and ,in ,according to from and Extract code snippets from and , , ,According to the same annotations of the code fragments, the code fragments of the source language program and the target program are matched one by one to obtain the fragment-level alignment data.

[0008] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned method for automatically constructing fragment-level alignment data are performed.

[0009] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned method for automatically constructing fragment-level alignment data are implemented.

[0010] The beneficial effects of the present invention are: The present invention utilizes a large model to gradually complete the automatic construction of fragment-level alignment data through multiple steps, and utilizes the large model to insert annotations into the source language program, which can ensure work efficiency in situations with a large number of languages. It also inserts the same numerical values ​​into the same code operation data of the target program based on the annotations of the source language program, ensuring that the source program and the target program can be aligned at the fragment level, so that the fragment-level alignment data constructed by the present invention is conducive to the model learning grammatical alignment knowledge. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A schematic diagram of a process for automatically constructing fragment-level alignment data provided in a specific embodiment; Figure 2 A schematic diagram of annotation insertion and annotation-based program rewriting in the source language program 1 provided in Example 1; Figure 3 Schematic diagram of data segmentation and matching between source language program 1 and target program 1 provided in Example 1; Figure 4 A schematic diagram of inserting comments into the source language program 2 provided in Example 2; Figure 5 A schematic diagram of program rewriting based on annotations provided in Example 2; Figure 6 Schematic diagram of data segmentation and matching of source language program 2 and target program 2 provided in Example 2. DETAILED DESCRIPTION

[0012] Specific implementation method 1: Figure 1 As shown, the steps of the method for automatically constructing fragment-level alignment data in this embodiment include: S1: Insert annotations into the source language program based on the large model; Input the source program into the big model, prompting the big model to insert natural language comments inside it, including: For a given source language program s , using a large model M Source language program s Insert comments at various types of code in the code. The position of the comment insertion is completely determined by The main principle is to insert as many comments as possible to ensure that the length of each code snippet is not too long and generate a new source language program. ,in, For the k Notes, For the k A code snippet, it should be noted that The length of can be 0, because in some cases, the first comment can be inserted at the beginning of the code. The length of other code snippets is not 0.

[0013] Through step S1, on the one hand, we can explain the functional meaning of each small piece of code, and on the other hand, it can serve as a separator for our cutting fragments.

[0014] S2: rewrite the target program based on the source language program after inserting the annotations, and generate annotations in the target program that are consistent with the content and order of the source language program; This embodiment inputs the source program output by S1 and the original target program into the large model, prompting the large model to rewrite the original target program according to the output of S1. For a given target program programming language set and the original target program set ,in Indicates the target programming languages, Indicates Original target program written, large language model, M Receiving new source language programs as well as As input, according to the new source language program Rewrite the target program by rewriting the comment content and comment order in the target program, and generate a new source language program in the target program Output the rewritten target program with the same content and order as the comments in , ,in, express No. A code snippet, Indicates the annotations, and then obtain a new set of target programs .

[0015] The goal of this rewriting implementation is to ensure that the target program also includes comments, and that the content and order of these comments are consistent with those in the source program. This rewriting requirement ensures that the target program is aligned with the source program at the segment level, providing a foundation for subsequent segment-level data matching.

[0016] S3: Split the source language program and the target program into several code snippets containing corresponding comments, and match the code snippets of the source language program and the target program one by one according to the comments of the code snippets to obtain snippet-level alignment data.

[0017] New source language program As target program , based on the target program Building a new target program collection ,in , for each pair of target programs and ,in ,according to from and Extract code snippets from and , , ,According to the same comments of the code fragments, the code fragments of the source language program and the target program are matched one by one to obtain the fragment-level alignment data.,If any mismatch in the number or content between the annotations is found, the pair of fragments is simply ignored and the matching process continues.

[0018] In addition, to achieve the above purpose, this embodiment also provides a computer device, including a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned fragment-level alignment data automatic construction method are performed.

[0019] In addition, to achieve the above-mentioned purpose, this embodiment further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned method for automatically constructing fragment-level alignment data are implemented.

[0020] In summary, the present invention accurately matches the code snippets in the source language program and the target program through the same annotation content, ensuring that the source program and the target program can be aligned at the fragment level, so that the fragment-level alignment data constructed by the present invention is conducive to the model learning grammatical alignment knowledge.

[0021] Example 1 Combine Figure 2 and Figure 3 To illustrate this embodiment, the following tests were performed to verify the accuracy of the fragment-level alignment data construction of the present invention: This embodiment uses Figure 2 Taking the source language program 1 and the target program 1 as an example, first, step 1 is performed to input the source program 1 into the large model, prompting the large model to insert natural language comments into it, and setting the data in the source language program 1. lo =1, calculate the target value n The cube root of the data is obtained by rounding it up to an integer. hi ,Will lo and hi As a double pointer, according to lo and hi Define the range of sum of cube numbers, where lo For the low byte, hi For the high byte, that is, insert the comment at the code "lo = 1hi = round(math.pow(n, 1 / 3))": # Initializetwo pointers, lo starting from 1 and hi set to the cube root of n; Set loop range , iterate over all lo、hi Combination, through the current cycle lo、 hi Calculate the current value of source language program 1 curr , that is, insert a comment at "while (lo<= hi):curr = (lo*lo*lo + hi*hi*hi)": # Use a while loop to check all possible pairs of cubesbetween lo and hi; Determine the current value of source language program 1 curr Is it equal to the target value? n If it is equal, then output it, if it is not equal, then repeat the cycle until the current value of the source language program is curr Equal to the target value n , that is, insert a comment at "if (curr == n): return True": # if the sum of cubes equals n, return True Then, proceed to step 2, input the annotated source language program 1 and target program 1 into the large model, rewrite the target program 1 according to the annotated source language program 1, and insert completely identical comments into the target program 1 code, as follows: Setting target program 1 lo =1, calculated by library function cbrt(n) hi The initial value of lo and hi As a double pointer, according to lo and hiDefine the range of cube number summation, that is, insert a comment consistent with the source language program at the code "long long lo = 1, hi =(long longint)cbrt(n)": # Initialize two pointers, lostating from 1 and hi set to the cube root of n; Set loop range , iterate over all lo、hi Combination, through the current cycle lo、 hi Calculate the current value of source language program 1 curr , that is, insert the following comment at "while (lo<= hi):curr = (lo*lo*lo + hi*hi*hi)" that is consistent with the source language program 1: # Use a while loop to check all possible pairs of cubes between lo and hi; Determine the current value of source language program 1 curr Is it equal to the target value? n If it is equal, then output it, if it is not equal, then repeat the loop until the current value of source language program 1 is curr Equal to the target value n , that is, insert a comment consistent with the source language program 1 at "if (curr == n): return True": # if the sum of cubes equals n, return True Finally, step 3 is performed to divide the source language program 1 and the target program 1 into code fragments containing unique comments. According to the comment content of each code fragment, the code fragments are matched one by one. The specific matching results are as follows: Figure 3 shown.

[0022] Depend on Figure 3 It can be seen that the present invention inserts the same numerical value into the same code operation data of the target program according to the annotations of the source language program, and can still maintain high work efficiency in situations with a large number of languages, and the code fragments are matched completely consistently, which is more conducive to the model learning grammatical alignment knowledge.

[0023] Example 2 Combine Figure 4-6 To illustrate this embodiment, the following tests were performed to verify the accuracy of the fragment-level alignment data construction of the present invention: This embodiment uses Figure 4Taking the code of the source language program 2 shown in the figure as an example, first, perform step 1 to input the source program 2 into the large model. First, insert the comment "# Import the math module to use mathematical functions like pow and cube root" at the code "import math", and insert the comment "# Function to check if a number can be expressed as the sum of two cubes" at the code "def sumOfTwoCubes(n)". Set the data in the source language program 1 to lo =1, calculate the target value n The cube root of the data is obtained by rounding it up to an integer. hi ,Will lo and hi As a double pointer, according to lo and hi Define the range of sum of cube numbers, where lo For the low byte, hi For the high byte, insert the comment " # Initialize two pointers, lostarting from 1 and hi set to the cube root of n" at the code "lo =1hi = round(math.pow(n, 1 / 3))".

[0024] Set loop range , iterate over all lo、hi Combination, through the current cycle lo、 hi Calculate the current value of source language program 1 curr, that is, insert the comment "# Use a while loop to check all possible pairs of cubes between lo and hi" at "while (lo<= hi):curr = (lo*lo*lo + hi*hi*hi)". When comparing the sum of the cubes with the target value n and then outputting, if the sum is less than n, increment lo to try a larger cube. Insert the comment "# If the sum of cubes equals n, return True" at the code "if (curr == n):return True". If the sum is greater than n, decrement hi to try a smaller cube. Insert the comment "# If the sum is greater than n, decrement hi to try a smaller cube" at the code "if (curr<n): lo += 1" If no such pair of numbers is found, return False. Insert the comment "# If no pair is found, return False" at the code " return False". Finally, test the function with N = 28 and output. That is, insert the comment "# Test the function with N = 28 and print the result" at the code "N =28if (sumOfTwoCubes(N)): print("True") else: print("False")".

[0025] such as Figure 5As shown, similarly, the large language model inserts the same comments into the corresponding type code of the target program 2 according to the source language program 2 after the annotations are inserted, specifically including: first, through<bits / stdc++.h> Contains common header files, introducing codes for mathematical functions such as power operations and cube roots, that is, in the code "#include<bits / stdc++.h> Using namespace std”, insert the same comment as in source language program 2 “Import the math moduleto use mathematical functions like pow and cube root”, introduce the function, check whether a number can be expressed as the sum of two cubes, that is, insert the same comment as in source language program 2 “Function to check if a number can be expressed as the sum oftwo cubes” at the code “bool sumOfTwoCubes(int n) {”, initialize two pointers, lo starting from 1, and hi set to the cube root of n, that is, insert the same comment as in source language program 2 “Initialize two pointers, lo starting from 1 and hi set to the cube root ofn” at the code “long longint lo = 1; hi = (long long int)cbrt(n);”

[0026] Then, use a while loop to check all possible pairs of cubes between lo and hi, i.e., insert the same comment as in Source Language Program 2 "Use a while loop to check all possible pairs of cubes between lo and hi" at the code "while (lo<= hi) {long long int curr = lo*lo*lo + hi*hi*hi". If the sum of the cubes equals n, return true, i.e., insert the same comment as in Source Language Program 2 "If the sum of cubes equals n, return True" at the code "if (curr == n) {return true;}". If the sum is less than n, increment lo to try a larger cube, i.e., insert the same comment as in Source Language Program 2 "If the sum is less than n, increment lo to try a larger cube" at the code " if (curr<n) { lo++;}". If the sum is greater than n, decrement hi to try a smaller cube, i.e., insert the same comment as in Source Language Program 2 "If the sum is greater than n, decrement hi to try a smaller cube" at the code " else {hi--;}}". If no pair is found, return false, then insert the same comment as in Source Language Program 2 "If no pair is found, return False" at the code "return false;}". Finally, test the function with N = 28 and print the result, i.e., insert the same comment as in Source Language Program 2 "Test the function with N = 28 and print the result" at the code "nt main() {int N = 28;if (sumOfTwoCubes(N)){cout<<"True";} else {cout<<"False";}return 0;}". Eventually, by rewriting the target program 2, at the corresponding type of code, insert comments that are the same in order and content as in Source Language Program 2 to facilitate subsequent code snippet division and code snippet matching.

[0027] Finally, according to the comments inserted in the first two steps, the code snippets in the source language program 2 and the target program 2 after the insertion of the comments are divided so that each code snippet has a unique comment, and the code snippets of the two programs are matched according to the comment content of the code snippets. The matching results are as follows: Figure 6 As shown by Figure 6 It can be seen that the present invention accurately matches the code snippets according to the annotation content, and constructs snippet-level alignment data based on this, providing a basis for the model to learn grammatical alignment knowledge.

[0028] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any technician familiar with the present profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical content disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement of the above embodiments made according to the technical essence of the present invention, within the spirit and principles of the present invention, without departing from the content of the technical solution of the present invention, shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A method for automatically constructing fragment-level alignment data, characterized in that: include: Step 1: Insert annotations into the source language program based on the big model; Step 2: Input the annotated source program and target program into the large model, rewrite the target program based on the annotated source program, and generate annotations in the target program that are consistent with the source program in content and order. Step 3: Split the source language program and the target program into several code snippets containing corresponding comments, and match the code snippets of the source language program and the target program one by one according to the comments of the code snippets to obtain snippet-level alignment data.

2. The method for automatically constructing segment-level alignment data according to claim 1, characterized in that: The source language program inserting comments in step 1 includes: For a given source language program s , using a large model M Source language program s Insert comments into various types of code to generate new source language programs ,in, For the k Notes, For the k code snippets.

3. The method for automatically constructing segment-level alignment data according to claim 2, characterized in that: Step 2 specifically includes: A set of programming languages ​​for a given target program and the original target program set ,in Indicates the target programming languages, Indicates Original target program written, large language model, M Receiving new source language programs as well as As input, according to the new source language program Rewrite the target program by rewriting the comment content and comment order in the target program, and generate a new source language program in the target program Output the rewritten target program with the same content and order as the comments in , ,in, express No. code snippets, Indicates the annotations, and then obtain a new set of target programs .

4. The method for automatically constructing segment-level alignment data according to claim 1, wherein: Step 3 specifically includes: New source language program As target program , based on the target program Building a new target program collection ,in , for each pair of target programs and ,in ,according to from and Extract code snippets from and , , ,According to the same annotations of the code fragments, the code fragments of the source language program and the target program are matched one by one to obtain the fragment-level alignment data.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images