A fully automatic test sample generation and evaluation method based on regular expressions

Abstract

The present invention discloses a fully automatic test sample generation and evaluation method based on regular expressions. The present invention firstly gives a set of regular expressions corresponding to the test samples according to the requirements of the question; and defines a formatting function; secondly, according to the set of regular expressions, a generation function is used to generate corresponding test samples, and a set of test samples is formatted by the formatting function, and multiple groups of test samples form a test sample list; then the test samples in the list are used to test the program submitted by the user and the standard program of the question respectively to obtain the output result; finally, a comparison function is used to determine whether the program submitted by the user passes the test of the test sample. The present invention uses regular expressions to randomly generate test samples, which can effectively solve the problem of test sample leakage. By generating test samples based on regular expressions, it can be achieved that the test samples used each time when judging the question are random, which effectively avoids some users from designing programs in a targeted manner.

Description

Technical Field

[0001] The present invention relates to the field of automatic testing, and in particular to a full-automatic test sample generation and evaluation method based on regular expressions. Background Art

[0002] Computer and information technology majors have higher requirements for the ability to combine programming theory with practice. Programming training platforms, also known as Online Judge platforms (online judging systems, hereinafter referred to as OJ), allow users to submit program source code online through access to the platform. The system compiles and executes the source code and judges questions through pre-designed test data, that is, verifies the correctness of the program source code. Such platforms provide users with a programming training environment to help them further consolidate their programming knowledge and improve their programming practice capabilities.

[0003] At present, there are many OJ platforms in my country, such as Zhejiang University's PTA platform, Likou platform, Luogu platform, and OJ platforms built by major universities. For programming training in computer and information technology majors, these OJ platforms provide students with a practical online programming training environment to help them further consolidate what they have learned in class and improve their practical skills.

[0004] Test samples are an extremely important part of online question judging. Currently, mainstream OJ platforms all use fixed test samples to verify the correctness of the source code, that is, specific test samples are specified when adding questions, and the test samples are used for judging questions thereafter. This method of judging questions brings some problems because the test samples are fixed, mainly including:

[0005] (1) Test sample leakage. Once a test sample is leaked, some users will design programs specifically to pass the test of these test samples instead of solving the problem fundamentally. This deviates from the original intention of using the OJ platform to cultivate students' computational thinking and improve their programming skills.

[0006] (2) It is difficult for users to locate errors in source code. Since test samples are fixed and cannot be provided to users at will to avoid test sample leakage, when the source code submitted by the user contains an error in the answer, the user can only debug his or her source code blindly and cannot quickly locate the problem, which may eventually cause the user to give up solving the problem or lose interest in programming. Summary of the invention

[0007] In order to make up for the deficiencies of the prior art, the present invention provides a fully automatic test sample generation and evaluation method based on regular expressions.

[0008] The technical solution adopted by the present invention is:

[0009] The present invention comprises the following steps:

[0010] 1) According to the requirements of the topic, give a set of regular expressions corresponding to the test samples;

[0011] 2) Define the formatting function format(X) to format the input data;

[0012] 3) When the user submits the source code, the generator function generate(r i ) generates corresponding test samples, and formats them using the formatting function format(X) to obtain a set of test sample inputs, and multiple sets of test sample inputs constitute a test sample list I;

[0013] The generating function generate(r i ) is used to convert the regular expression r i Convert to the corresponding string, the function returns a string that matches the regular expression r i The string to match;

[0014] 4) Use the test examples in List I to test the program P′ submitted by the user and the standard program P of the question, and obtain the output results O′ and O respectively;

[0015] Let output=P(input), input∈I, output′=P′(input), input∈I,

[0016] have to:

[0017] 5) Use the comparison function compare(output, output′) to compare whether the two sets of outputs output and output′ are consistent. If they are the same, the comparison function returns TRUE, indicating that the program P′ passes the test of this set of test samples; if the comparison function returns FALSE, it means that the program P′ cannot pass the test of this set of test samples;

[0018] If the program P′ can pass the test of all test samples in I, then P′ is correct, otherwise P′ is wrong;

[0019] At this point, the fully automatic generation and evaluation of test samples has been achieved.

[0020] Beneficial effects of the present invention:

[0021] First, the present invention uses regular expressions to randomly generate test samples, which can effectively solve the problem of test sample leakage. Based on regular expressions to generate test samples, the test samples used in each judgment can be random, which effectively avoids some users from designing programs in a targeted manner.

[0022] Secondly, since regular expressions have very strong matching capabilities, the present invention is technically feasible. In addition, the test samples used in each judgment are random rather than fixed, so when the user submits a program error, the corresponding test sample can be provided to the user, so that the user can locate the program error by himself, avoiding blind debugging, which reflects the practicality of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, a brief introduction is given below to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 The flowchart of the fully automatic test sample generation and evaluation method of the present invention is shown. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments, and are not intended to limit the scope of the present invention.

[0026] The overall concept of the present invention: The present invention generates test data based on the form of specifying test samples based on regular expressions, uses the generated test data to determine the correctness of source code submitted by users, and provides a new judgment method for an online judgment platform.

[0027] Example

[0028] A fully automatic test sample generation and evaluation method based on regular expressions, such as Figure 1 As shown, the specific steps include:

[0029] 1. According to the requirements of the question, give the regular expression set R corresponding to the test sample. R can contain multiple different regular expressions r i (1≤i≤n), that is In addition, the standard procedure P for this problem is also given.

[0030] Specifically, each question has a corresponding input, and a corresponding regular expression can be given according to the input. For example, for a question about adding two numbers, the input must be all numbers, and a regular expression can be specified as [\d]{1,20}, which means generating a string of all numbers with a length of 1 to 20 digits.

[0031] Then a standard program P is given so that for every set of inputs that meet the requirements of the question, a set of correct output results can be obtained.

[0032] 2. Define the formatting function format(X) to format the input data.

[0033] Specifically, according to the description of the question, we can get the format requirements of a test sample. For example, two data in each line, each data separated by a space, a total of 2 lines, and construct the formatting function format(X) based on this, the independent variable X is the unformatted input data set, and the function return value is a set of input data that has completed formatting.

[0034] 3. When the user submits the source code, the generator function generate(r i )(1≤i≤n) generates corresponding test samples and formats them with the formatting function format(X) to obtain a set of test sample inputs. Multiple sets of test sample inputs constitute a test sample list I.

[0035] Right now,

[0036] Generate function is used to convert the regular expression r i Convert to the corresponding string, the function returns a string that matches the regular expression r i It should be noted that a regular expression may match multiple different strings, but the generate function only returns one of them. i , calling the generate function multiple times may not return exactly the same value.

[0037] Specifically, the regular expression in this example is r = [\d] {1, 20}, and the generating function generate(r i) generates a series of data and forms a set X. For example, a possible set X = {'12', '100', '50', '900'}. Use the formatting function format(X) to get a set of test samples input = '12 100\n50 900'. For the convenience of description, the test data of this embodiment has only one set of test samples, that is, the test sample set I = [input,] = ['12100\n50 900',].

[0038] 4. Use the test examples in List I to test the program P′ submitted by the user and the standard program P in the question, and get the output results O′ and O respectively.

[0039] Since programs have input and output, programs P and P' can be considered as functions. Let output = P(input), input∈I, where P is a standard program, input is a set of test samples in list I, and output is the output corresponding to the set of test samples. Similarly, output' = P'(input), input∈I.

[0040] Therefore, we can get:

[0041] Specifically, the standard program P(input) is used to obtain the correct output result output = '112\n950', and all outputs are sequentially composed of an O = ['112\n950',]. The program P'(input) submitted by the user is used to obtain an output result output', and all output' are sequentially composed of an O'.

[0042] 5. Use the comparison function compare(output, output') to compare whether the two sets of output output and output' are consistent. If they are the same, the comparison function compare returns TRUE, indicating that the program P' passes the test of this set of test samples; if the comparison function compare returns FALSE, it means that the program P' cannot pass the test of this set of test samples. If the program P' can pass the test of all test samples in I, then P' is correct, otherwise P' is wrong. That is:

[0043]

[0044]

[0045] Specifically, in this embodiment, if the program output result submitted by the user is output'='112\n950', the comparison function compare(output, output') returns TRUE, indicating that the answer to this group of test samples is correct. Since this example has only one group of test samples, the answer to this question is correct. If the program output result submitted by the user is output'='102\n950', the comparison function compare(output, output') returns FALSE, and the answer to this question is incorrect.

Claims

1. A fully automatic test sample generation and evaluation method based on regular expressions, characterized in that The following steps are involved: 1) According to the requirements of the topic, give a set of regular expressions corresponding to the test samples; 2) Define the formatting function format(X) to format the input data; 3) When the user submits the source code, the generator function generate(r i ) generates corresponding test samples, and formats them using the formatting function format(X) to obtain a set of test sample inputs, and multiple sets of test sample inputs constitute a test sample list I; The generating function generate(r i ) is used to convert the regular expression r i Convert to the corresponding string, the function returns a string that matches the regular expression r i The string that matches; 4) Use the test examples in List I to test the program P′ submitted by the user and the standard program P of the question, and obtain the output results O′ and O respectively; Let output=P(input), input∈I, output′=P′(input), input∈I, have to: 5) Use the comparison function compare(output, output′) to compare whether the two sets of outputs output and output′ are consistent. If they are the same, the comparison function returns TRUE, indicating that the program P′ passes the test of this set of test samples; if the comparison function returns FALSE, it means that the program P′ cannot pass the test of this set of test samples; If the program P′ can pass the test of all test samples in I, then P′ is correct, otherwise P′ is wrong; At this point, the fully automatic generation and evaluation of test samples has been achieved.

2. The method for generating and evaluating fully automatic test samples based on regular expressions according to claim 1, characterized in that: The regular expression set includes multiple different regular expressions.

3. The method for generating and evaluating fully automatic test samples based on regular expressions according to claim 1, characterized in that: When a set of regular expressions corresponding to the test examples is given, the standard procedure P for the question is also given.

Images