Unit test generation method based on large language model

Abstract

The invention relates to a unit test generation method based on a large language model, which comprises the following steps: collecting code fine tuning data to obtain an original data set; preprocessing the original data set to obtain an optimized data set; generating unit test data according to a query problem and a code solution in the optimized data set; training a large language model based on supervision fine tuning through the optimized data set and the unit generation data to obtain a unit test generator; and generating a unit test result of the to-be-tested program through the unit test generator. The efficiency, accuracy and flexibility of the unit test generation method can be improved, and support is provided for quality assurance and test optimization in the software development process.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a unit test generation method based on a large language model. Background Art

[0002] Large Language Models (LLMs) have shown significant potential in code generation tasks, but their ability to generate correct answers in a single reasoning is still limited, especially in scenarios that require complex logical reasoning or precise verification. Current mainstream technologies usually use the Best-of-N strategy to improve generation quality, that is, by generating candidate answers multiple times and using a verifier or reward model to screen out the best result.

[0003] Existing research usually uses unit tests as verifiers to screen candidate codes. Specifically, for specific problems and codes, first use the large language model to generate test cases, including the input parameters and expected outputs of the tested functions; then, based on the results of running the code on the test cases, select the program with the best performance. However, this method faces many challenges in practical applications. The most important problem is that due to the inherent "hallucination" phenomenon of the large language model, the unit tests generated by it may not be reliable, and the expected output and input parameters are often inconsistent, which weakens the effectiveness of the verifier in screening the optimal program. Summary of the invention

[0004] The present invention provides a unit test generation method based on a large language model, so as to solve the defects of the prior art.

[0005] The present invention provides a unit test generation method based on a large language model, comprising:

[0006] S1: Collect code fine-tuning data and obtain the original dataset;

[0007] S2: preprocessing the original data set to obtain an optimized data set;

[0008] S3: Generate unit test data according to the query problems and code solutions in the optimized data set;

[0009] S4: training the large language model based on supervised fine-tuning using the optimized data set and the unit generated data to obtain a unit test generator;

[0010] S5: Generate unit test results for the program to be tested by the unit test generator.

[0011] According to a unit test generation method based on a large language model provided by the present invention, in step S1, code fine-tuning data is collected from the CodeFeedback-Filtered-Instruction dataset and the TACO dataset.

[0012] According to a unit test generation method based on a large language model provided by the present invention, step S2 further includes:

[0013] S21: Eliminate problematic data from the original data set to obtain a filtered data set;

[0014] S22: Formatting the code solutions in the filtered data set into a function form to obtain an optimized data set.

[0015] According to a unit test generation method based on a large language model provided by the present invention, the problem data in step S21 includes random data, non-deterministic data, and missing test standard data.

[0016] According to a unit test generation method based on a large language model provided by the present invention, step S3 further includes:

[0017] S31: Using the Llama3.1-70B model, multiple samplings are performed on the query problems and code solutions in the optimized data set to obtain basic unit test data;

[0018] S32: verifying the basic unit test data and removing the wrong unit test data to obtain filtered unit test data;

[0019] S33: Distinguish the filtered unit test data and remove erroneous unit test data to obtain unit test data.

[0020] According to a unit test generation method based on a large language model provided by the present invention, step S32 further includes:

[0021] S321: Apply the basic unit test data to the code solution in the optimized data set to obtain an execution result;

[0022] S322: judging the corresponding basic unit test data based on the execution result, when the execution result allows the code solution to pass, the corresponding basic unit test data is correct basic unit test data, and when the execution result cannot allow the code solution to pass, the corresponding basic unit test data is incorrect basic unit test data;

[0023] S323: Eliminate the erroneous basic unit test data in the basic unit test data to obtain filtered unit test data.

[0024] According to a unit test generation method based on a large language model provided by the present invention, step S322 also includes:

[0025] By capturing the execution feedback information generated by the Python interpreter, the error-based unit test data can be repaired.

[0026] According to a unit test generation method based on a large language model provided by the present invention, step S33 further includes:

[0027] S331: Generate error code solution;

[0028] S332: applying the filtering unit test data to the error code solution; when the filtering unit test data does not distinguish the error code solution, the corresponding filtering unit test data is incorrect filtering unit test data; when the filtering unit test data distinguishes the error code solution, the corresponding filtering unit test data is correct filtering unit test data;

[0029] S333: Eliminate the erroneous filtering unit test data in the filtering unit test data to obtain unit test data.

[0030] According to a unit test generation method based on a large language model provided by the present invention, the unit test generator in step S4 takes the query problem and code solution in the optimized data set as input and takes unit test data as output.

[0031] According to a unit test generation method based on a large language model provided by the present invention, the loss function of the unit test generator in step S4 is expressed as:

[0032]

[0033] Among them, L(x,y) is the loss function of the unit test generator, x is the data in the input optimization data set, y is the unit test result generated by the unit test generator, i is the index value of the obtained token, j is the index value of the given token, Model(y i |x,y j<i) ) is the probability of generating the i-th token of y given the j=i-1-th token of x and y.

[0034] The present invention provides a unit test generation method based on a large language model. By collecting and preprocessing code fine-tuning data, unit test data that is closer to the actual programming scenario can be generated. At the same time, the large language model is used for supervised fine-tuning, which can significantly improve the accuracy and efficiency of unit test generation, so that the generated unit test can better reflect the actual function and potential problems of the code. In the data preprocessing stage, the present invention ensures the high quality of the large language model data set used for training and optimization by excluding random, non-deterministic and problematic data lacking test standards, which helps to reduce the noise and misjudgment of the model when generating unit tests, and improves the reliability and effectiveness of the generated unit tests. Secondly, the present invention can generate unit test data by multiple sampling and verification of erroneous unit test data, and repairing erroneous basic unit test data. More targeted and discriminative unit test data, which can not only verify the correctness of the code, but also effectively distinguish the solutions to the error codes, thereby improving the coverage and effectiveness of software testing; in addition, the present invention trains a large language model based on the optimized data set and the unit generation data, and the obtained unit test generator has a stronger generalization ability, and can generate unit test data that meets the requirements for different types of codes and test requirements, further improving the level of automated testing in the software development process; the unit test generation method of the present invention realizes the automation and intelligence of the software testing process, reduces the workload of testers, and improves the testing efficiency. At the same time, by continuously optimizing and improving the unit test generation method, the software development process can be promoted to a higher level of automation and intelligence. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 A schematic diagram of a flow chart of a unit test generation method based on a large language model provided by an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of a flow chart of a method for obtaining an optimized data set provided by an embodiment of the present invention;

[0038] Figure 3 A schematic diagram of a unit test data acquisition method flow chart provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments, and they should not be understood as limitations on the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In the description of the present invention, it should be understood that the terms used are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0040] The embodiments of the present invention are described below with reference to the accompanying drawings.

[0041] like Figure 1 As shown, the present invention provides a unit test generation method based on a large language model, comprising:

[0042] S1: Collect code fine-tuning data and obtain the original dataset.

[0043] In step S1, the existing high-quality code fine-tuning data is first used as the core foundation of the entire process, and a strictly screened data set is selected.

[0044] In step S1, code fine-tuning data is collected from the CodeFeedback-Filtered-Instruction dataset and the TACO dataset.

[0045] Furthermore, the collected data sets include training sets of CodeFeedback-Filtered-Instruction and TACO data sets. The data sets are designed to provide stable and efficient support for code generation and instruction tuning tasks, thereby ensuring that the performance of the model in various complex scenarios can meet or exceed expectations.

[0046] As the basic part of the unit testing method of the present invention, CodeFeedback-Filtered-Instruction is a highly optimized and precisely constructed code instruction query set, and the data comes from four well-known open source code instruction fine-tuning datasets. These datasets are obtained by cleaning, integrating and deeply optimizing a large number of open source code resources, specifically including Magicoder-OSS-Instruct, Magicoder-Evol-Instruct, Evol-Instruct-Code and high-quality subsets related to Python code in ShareGPT. The CodeFeedback-Filtered-Instruction dataset ensures that each instruction has a high degree of relevance, accuracy and applicability, providing a reliable foundation for the code generation and instruction tuning of the present invention.

[0047] At the same time, the TACO dataset provides another important data source for the present invention that focuses on algorithm code generation. The dataset integrates a large number of programming problems from multiple well-known programming platforms (such as LeetCode and Codeforces), covering not only traditional algorithmic questions, but also more complex and innovative programming scenarios. Through in-depth mining and integration of these problems, the TACO dataset can provide a variety of training materials for the code generation model, thereby significantly improving the accuracy, stability and versatility of the model in solving algorithmic problems.

[0048] S2: Preprocessing the original data set to obtain an optimized data set.

[0049] In step S2, a large language model is used to filter out problems that are not suitable for unit test generation (for example, problems involving random number generation). Then, the code program is formatted to convert the non-standard code in the original data set into a functional form to facilitate the generation and processing of subsequent unit tests.

[0050] like Figure 2 As shown, step S2 further includes:

[0051] S21: Eliminate problematic data from the original data set to obtain a filtered data set.

[0052] In order to optimize the quality of the data, the present invention adopts a principle-based screening method in step S21 to refine and filter the data set at a higher level. Specifically, the present invention uses the Llama3.1-70B-Instruct model to accurately identify and eliminate data that is not suitable for unit testing. For example, problems involving randomness, non-determinism, or lack of clear test standards will be systematically excluded. This principle-driven screening process not only ensures the high testability and consistency of the data, but also lays a solid reliability foundation for the subsequent test generation link.

[0053] The problem data in step S21 includes random data, non-deterministic data, and missing test standard data.

[0054] Furthermore, random data refers to data that contains random elements or relies on random processes to generate data. In the context of unit testing, random data may lead to inconsistent and unpredictable test results. For example, if the code solution contains a random number generator and the test does not have proper control or expectations for these random elements, then different results may be obtained each time the test is run, making the test unable to stably verify the correctness of the code.

[0055] Non-deterministic data is similar to random data, but the difference is that non-deterministic data does not rely on random processes, but may be affected by external factors such as system status, timestamps, network requests, etc. These factors are usually difficult to control or predict in unit testing, so they may lead to inconsistent test results. Non-deterministic data makes tests difficult to repeat, thereby reducing the reliability of the test.

[0056] Missing test criteria data refers to data that does not have clearly defined test criteria or expected results. In unit testing, each test should have a clear expected result so that it can be verified that the code works as expected. If the test data does not provide these criteria or results, then the test cannot effectively evaluate the correctness of the code. This situation may be caused by incomplete test descriptions, poorly designed test data, or unclear test requirements.

[0057] S22: Formatting the code solutions in the filtered data set into a function form to obtain an optimized data set.

[0058] After the data screening is completed in step S21, the present invention further performs a comprehensive functional reconstruction of the original code solution. Through this process, the structure of the original code is converted into a standardized function form. This conversion not only improves the modularity and readability of the code, but also significantly enhances its applicability in the unit test generation task. The core goal of code functional reconstruction is to facilitate the subsequent automated unit test generation, so that each section of code can be parsed and tested in a consistent form, thereby greatly improving the efficiency and accuracy of unit test generation.

[0059] S3: Generate unit test data according to the query problems and code solutions in the optimized data set.

[0060] In step S3, based on the code data preprocessed in step S2, unit test data is generated using a large language model. To ensure the correctness of the code program, the present invention also uses the correct code program already existing in the data set as a verifier to verify whether the generated unit test meets expectations. If there is an error in the generated unit test, the incorrect unit test is repaired through the return status of the code executor combined with the large language model. In addition, to further improve the quality of unit testing, the present invention will filter out false positive test cases, that is, those unit tests that the erroneous program can pass, thereby ensuring that the generated unit test is more reliable.

[0061] like Figure 3 As shown, step S3 further includes:

[0062] S31: Using the Llama3.1-70B model, multiple samplings are performed on the query problems and code solutions in the optimized data set to obtain basic unit test data.

[0063] In step S31, based on the query and code solutions provided in the dataset, the present invention uses the Llama3.1-70B model to generate diversified unit tests through multiple samplings. This generation strategy aims to expand the coverage of unit tests, thereby improving the verification capability and robustness of the code solution.

[0064] S32: Verify the basic unit test data to remove erroneous unit test data, and obtain filtered unit test data.

[0065] Wherein, step S32 further comprises:

[0066] S321: Apply the basic unit test data to the code solution in the optimized data set to obtain an execution result.

[0067] S322: The corresponding basic unit test data is judged based on the execution result. When the execution result allows the code solution to pass, the corresponding basic unit test data is correct basic unit test data. When the execution result cannot allow the code solution to pass, the corresponding basic unit test data is incorrect basic unit test data.

[0068] After generating the unit tests in step S31, the present invention further performs execution verification on these tests. Specifically, the present invention applies each unit test to the ground truth code solution provided in the data set and records its execution results, which are used as the basis for the true / false label of the unit test: the unit test that can successfully pass the ground truth code solution is considered to be a correct unit test, and the unit test that fails is marked as an incorrect unit test.

[0069] Specifically, each generated unit test is first applied to the real code solution provided in the dataset to verify the effectiveness and accuracy of the unit test. After executing the unit test, the results of each test are recorded. These results will be used to evaluate the correctness of the unit test.

[0070] Follow-up judgment, correct unit testing: If the unit test allows the real code solution to pass successfully, that is, no errors or exceptions are caused, then the unit test is considered correct, which means that a part of the functionality of the code is correctly verified.

[0071] Faulty unit tests: If a unit test fails the real code solution, for example, because the expected result of the test does not match the actual result, then the unit test is marked as faulty, indicating that there is a logical error in the test itself, or a defect in the part of the code it tests.

[0072] Wherein, step S322 also includes:

[0073] By capturing the execution feedback information generated by the Python interpreter, the error-based unit test data can be repaired.

[0074] For some difficult code instructions, the present invention often consumes a lot of computing resources to generate a small number of correct unit tests. These complex instructions not only have higher requirements for test generation, but also rely solely on multiple sampling methods with low efficiency, which is difficult to meet the needs of efficient generation.

[0075] In order to collect more correct unit tests more efficiently when dealing with difficult problems, the present invention introduces an optimization method based on Python interpreter execution feedback. For erroneous unit tests, the present invention captures the execution feedback information generated by the Python interpreter and uses these feedbacks to guide the repair of unit tests. Specifically, the present invention uses the Llama3.1-70B model again to repair erroneous unit tests according to these execution feedbacks. This repair method based on execution feedback is more efficient than multiple sampling strategies and can significantly reduce the time and resources required to generate correct unit tests.

[0076] S323: Eliminate the erroneous basic unit test data in the basic unit test data to obtain filtered unit test data.

[0077] S33: Distinguish the filtered unit test data and remove erroneous unit test data to obtain unit test data.

[0078] Wherein, step S33 further comprises:

[0079] S331: Generate error code solution.

[0080] S332: Apply the filtering unit test data to the error code solution. When the filtering unit test data does not distinguish the error code solution, the corresponding filtering unit test data is incorrect filtering unit test data. When the filtering unit test data distinguishes the error code solution, the corresponding filtering unit test data is correct filtering unit test data.

[0081] S333: Eliminate the erroneous filtering unit test data in the filtering unit test data to obtain unit test data.

[0082] After step S32, in order to select high-quality unit tests from all correct unit tests, the present invention also believes that high-quality unit tests can not only enable correct code solutions to pass the test successfully, but also exclude incorrect code solutions as much as possible. Based on this principle, the present invention introduces an additional filtering process: first, a model with weaker performance is used to generate incorrect code solutions (incorrect solutions), and then the discrimination ability of unit tests is tested through these incorrect code solutions. "False positive unit tests" that cannot exclude incorrect code solutions, that is, unit tests that fail to reject incorrect solutions, will be further filtered out.

[0083] Specifically, in step S33, the present invention uses a model with weaker performance than Llama3.1-70B used to generate unit tests to generate a series of erroneous code solutions, which contain logical errors, syntax errors, performance issues or other types of defects; subsequently, the correct unit tests generated previously are applied to these erroneous code solutions to observe whether these unit tests can correctly identify and reject these erroneous codes. If a unit test fails to reject one or more erroneous code solutions (that is, it mistakenly believes that these erroneous codes are correct), then the unit test is regarded as a "false positive unit test". These tests may not be able to effectively distinguish between correct and incorrect code solutions, and therefore will be further filtered out.

[0084] S4: Using the optimized data set and the unit generation data, the large language model is trained based on supervised fine-tuning to obtain a unit test generator.

[0085] The unit test generator in step S4 takes the query problem and code solution in the optimized data set as input and takes unit test data as output.

[0086] The loss function of the unit test generator in step S4 is expressed as:

[0087]

[0088] Among them, L(x, y) is the loss function of the unit test generator, x is the data in the input optimization data set, y is the unit test result generated by the unit test generator, i is the index value of the obtained token, j is the index value of the given token, Model(y i |x,y j <i)) is the probability of generating the i-th token of y given the j=i-1-th token of x and y.

[0089] In step S4, the present invention uses supervised fine-tuning (SFT) to train the unit test generator. In the process of constructing training data, the present invention uses the problem description and code solution as the input instructions of the model, and uses high-quality unit tests as the target output of the model. This process aims to enable the model to more accurately generate unit tests for specific code solutions through supervised learning, thereby improving the overall performance and applicability of the unit test generator.

[0090] S5: Generate unit test results for the program to be tested by the unit test generator.

[0091] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0092] A unit test generation method based on a large language model of the present invention is described below in conjunction with a specific embodiment.

[0093] In order to evaluate the generalization of the present invention in multiple code program generation models, the present invention selected multiple open source and closed source models of different scales for testing, including Llama3-8B, Llama3-70B, GPT-3.5-turbo and GPT-4o-mini.

[0094] In order to reflect the performance advantages of the present invention, the present invention selects multiple baseline methods using unit tests as verifiers, including MBR-Exec based on minimum Bayesian risk decoding, CodeT based on dual execution agreement, and MPSC based on multi-angle consistency.

[0095] The present invention selects the HumanEval Plus dataset as the evaluation benchmark. The HumanEval Plus dataset is a well-known code generation dataset, including 164 manually written data and detailed and accurate unit tests, which are used to evaluate the correctness of the code program generated by the model.

[0096] Table 1 Accuracy results on multiple code generation models

[0097]

[0098] The present invention conducts repeated experiments on the HumanEval Plus dataset and involves multiple code program generation models. For each data point, 100 candidate code programs and 100 unit tests are generated. For the unit test generation of the present invention, a standard majority voting method is used to compare with the baseline method, and the results are shown in Table 1.

[0099] As can be seen from Table 1, for multiple open source and closed source models of different scales, including Llama3-8B, Llama3-70B, GPT-3.5-turbo and GPT-4o-mini, the accuracy of the present invention is the highest, and its performance is stronger than that of the baseline method.

[0100] In addition, in order to evaluate the quality of the unit test generated by the model after training, the present invention compares and analyzes the unit test generated by the model before training in terms of accuracy. The specific results are shown in Table 2.

[0101] Table 2 Unit test quality

[0102]

[0103] As can be seen from Table 2, the unit tests generated by the model trained by the present invention have higher accuracy and higher quality than the Llama3.1-8B model, which proves the high performance and effectiveness of the present invention.

[0104] In general, the unit test generation method based on a large language model of the present invention designs an automated synthetic data framework, utilizes existing code instruction fine-tuning data to generate high-quality unit tests, and then uses the coding problems, program code and synthesized high-quality unit test data in the code instruction fine-tuning data as training data to train a high-quality unit test generation model.

[0105] First, based on the automatic data synthesis framework, the present invention can use existing code instructions to fine-tune data and generate higher-quality unit test data; second, by training a high-quality unit test generation model, the present invention significantly improves the reliability and quality coverage of the generated unit tests. Experimental results show that the unit test generation model trained by the present invention has a significantly better generation accuracy than the existing baseline method.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A unit test generation method based on a large language model, characterized in that: include: S1: Collect code fine-tuning data and obtain the original dataset; S2: preprocessing the original data set to obtain an optimized data set; S3: Generate unit test data according to the query problems and code solutions in the optimized data set; S4: training the large language model based on supervised fine-tuning using the optimized data set and the unit generated data to obtain a unit test generator; S5: Generate unit test results for the program to be tested by the unit test generator.

2. The unit test generation method based on a large language model according to claim 1, characterized in that: In step S1, code fine-tuning data is collected from the CodeFeedback-Filtered-Instruction dataset and the TACO dataset.

3. The unit test generation method based on a large language model according to claim 1, characterized in that: Step S2 further comprises: S21: Eliminate problematic data from the original data set to obtain a filtered data set; S22: Formatting the code solutions in the filtered data set into a function form to obtain an optimized data set.

4. The unit test generation method based on a large language model according to claim 3, characterized in that: The problem data in step S21 includes random data, non-deterministic data, and missing test standard data.

5. The unit test generation method based on a large language model according to claim 1, characterized in that: Step S3 further comprises: S31: Using the Llama3.1-70B model, multiple samplings are performed on the query problems and code solutions in the optimized data set to obtain basic unit test data; S32: verifying the basic unit test data and removing the wrong unit test data to obtain filtered unit test data; S33: Distinguish the filtered unit test data and remove erroneous unit test data to obtain unit test data.

6. A unit test generation method based on a large language model according to claim 5, characterized in that: Step S32 further includes: S321: Apply the basic unit test data to the code solution in the optimized data set to obtain an execution result; S322: judging the corresponding basic unit test data based on the execution result, when the execution result allows the code solution to pass, the corresponding basic unit test data is correct basic unit test data, and when the execution result cannot allow the code solution to pass, the corresponding basic unit test data is incorrect basic unit test data; S323: Eliminate the erroneous basic unit test data in the basic unit test data to obtain filtered unit test data.

7. A unit test generation method based on a large language model according to claim 6, characterized in that: Step S322 also includes: By capturing the execution feedback information generated by the Python interpreter, the error-based unit test data can be repaired.

8. The unit test generation method based on a large language model according to claim 5, characterized in that: Step S33 further comprises: S331: Generate error code solution; S332: applying the filtering unit test data to the error code solution; when the filtering unit test data does not distinguish the error code solution, the corresponding filtering unit test data is incorrect filtering unit test data; when the filtering unit test data distinguishes the error code solution, the corresponding filtering unit test data is correct filtering unit test data; S333: Eliminate the erroneous filtering unit test data in the filtering unit test data to obtain unit test data.

9. The unit test generation method based on a large language model according to claim 1, characterized in that: The unit test generator in step S4 takes the query problem and code solution in the optimized data set as input and takes unit test data as output.

10. A unit test generation method based on a large language model according to claim 1, characterized in that: The expression of the loss function of the unit test generator in step S4 is: Among them, L(x,y) is the loss function of the unit test generator, x is the data in the input optimization data set, y is the unit test result generated by the unit test generator, i is the index value of the obtained token, j is the index value of the given token, Model(y i |x,y j<i) ) is the probability of generating the i-th token of y given the j=i-1-th token of x and y.

Images