A method, device and medium for multi-source code data processing and large language model fine-tuning training

Abstract

The present invention relates to a method, device, and medium for processing multi-source code data and fine-tuning a large language model. The data processing method includes the following steps: obtaining a first open source code dataset and performing instruction evolution to obtain expanded code data, which is then integrated with a second open source code dataset to obtain multi-source code data; filtering and deduplicating the multi-source code data to obtain filtered multi-source code data; extracting code comprehension task data according to a data construction process; designing data-specific prompt generation instructions, using the filtered multi-source code data as basic data to generate data-specific prompts, and inserting the generated prompts into the beginning of the corresponding original basic data instructions to form data-specific prompt customized data; and merging the code comprehension task data and the data-specific prompt customized data into the filtered multi-source code data to achieve data reconciliation. Compared with the prior art, the present invention has the advantages of good data diversity, reconciling inherent conflicts in multi-source code data, and strong instruction-following capabilities of the trained model.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device and medium for multi-source code data processing and large language model fine-tuning training. Background Art

[0002] Large Language Models (LLMs) acquire code capabilities through pre-training and fine-tuning on code data, and have shown great potential in tasks such as code generation, mathematical reasoning, and tool calling.

[0003] Existing open-source work has accumulated a large amount of code data for pre-training, and recent code fine-tuning models have demonstrated the effectiveness of high-quality code fine-tuning datasets. For example, existing large language model fine-tuning methods for code data, such as WizardCoder (see "WizardCoder: Empowering Code Large Language Models with Evol-Instruct") and Magicoder (see "Magicoder: Source code is all you need"), also aim to construct high-quality code data to improve the code capabilities of large language models. However, these methods suffer from the following drawbacks:

[0004] a) The data sources for code fine-tuning of both are single and the data styles are limited;

[0005] b) Although both proposed methods to enhance the diversity of programming questions, they are still limited to a single type of code programming question answering data.

[0006] That is, in order to obtain better code capabilities, existing large language models are usually fine-tuned on a single specific type of code question and answer dataset. In other words, the existing methods of constructing code data for large language model fine-tuning often rely on a single code dataset of the programming question and answer type. The lack of diversity cannot fully stimulate the capabilities of the base model, which limits the performance, generalization and robustness of the fine-tuned large language model.

[0007] However, due to inherent conflicts such as varying quality and diverse styles within multiple source code data, the performance gains from directly mixing source code data for fine-tuning are suboptimal. For example, some existing large language model training techniques also integrate multi-source data, but only perform macro-level processing such as data filtering and ratio adjustments on the mixed data, without addressing the detailed conflicts within the mixed data. Summary of the Invention

[0008] The purpose of the present invention is to provide a method, device and medium for processing multi-source code data and fine-tuning training of a large language model, so as to process internal conflicts in mixed data, construct and reconcile multi-source code data, enrich data diversity, better stimulate the capabilities of the base model, and improve the performance, generalization and robustness of the large language model.

[0009] The purpose of the present invention can be achieved by the following technical solutions:

[0010] A method for processing multi-source code data includes the following steps:

[0011] Open source data evolution and integration: obtaining a first open source code dataset and using it as base data, performing instruction evolution on the base data using the first language model as a generative model to obtain expanded code data, and integrating the expanded code data with a second open source code dataset to obtain multi-source code data, wherein the multi-source code data includes instructions and responses;

[0012] Data filtering and deduplication: Filter and dedupe the data that does not meet the requirements in the multi-source code data to obtain the filtered multi-source code data;

[0013] Code comprehension task construction: Extract code comprehension task data according to the data construction process, which includes instruction evolution and data filtering and deduplication process;

[0014] Data-specific prompt generation: Design data-specific prompt generation instructions, use the filtered multi-source code data as the basic data, use the second largest language model as the generation model to generate data-specific prompts, and insert the generated prompts into the beginning of the corresponding original basic data instructions to form data-specific prompt customized data;

[0015] Data merging: Merge code comprehension task data and data-specific prompt customized data into the filtered multi-source code data to achieve data reconciliation.

[0016] The data filtering and deduplication specifically involves filtering out data that meets any of the following conditions:

[0017] a) The length of the response in the multi-source code data is less than the preset threshold and does not contain the code;

[0018] b) The responses in multiple source code data contain code but fail to compile or pass test cases;

[0019] c) The similarity between the multiple source code data and samples in the predetermined sample set is greater than a preset threshold.

[0020] The similarity between the multiple source code data and samples in the predetermined sample set is greater than a preset threshold specifically means that any one of the N-gram similarity, the cosine similarity of the code segment embedding, or the similarity of the code segment syntax tree between the multiple source code data and the samples in the predetermined sample set exceeds the preset threshold.

[0021] The code comprehension task data includes:

[0022] 1) Instruction evolution task data: Data generated during the evolution and integration of open source data that meets any of the following conditions is selected as instruction evolution task data:

[0023] The instructions before and after instruction evolution are both present in the filtered multi-source code data;

[0024] The number of syntax tree levels of the reply code segment corresponding to the instruction after the instruction evolution is not less than the number of syntax tree levels of the reply code segment corresponding to the instruction before the evolution;

[0025] 2) Data filtering task data: Select low-quality data to construct multiple-choice question format data as data filtering task data, wherein the low-quality data includes:

[0026] The length of the response is less than the preset threshold and does not contain data with a code;

[0027] Data on code compilation failures;

[0028] The frequency of single-letter variables in the code is greater than the preset threshold and does not contain comments;

[0029] Failure to organize the code into data in the functional form required by the instructions;

[0030] 3) Code review task data: Design code review task data generation instructions, use multiple source code data as base data and the second largest language model as a generation model to generate code review task data, wherein the code review task data generation instructions meet the following requirements:

[0031] Professionalism: The instruction sets act as professional code reviewers to review the given code content;

[0032] Review content: The review content of the instruction settings includes correctness and clarity. Correctness means that the code has no syntactic and logical errors and implements the given function name. Clarity means that variables should have meaningful names and function parameters and return values should have type annotations.

[0033] Scoring criteria: Instructions set a preset score range, score the code based on correctness and clarity, and provide review comments;

[0034] Suggestions for modification: The directive sets requirements to provide suggestions for further improvement;

[0035] Optimize code: The instruction sets the requirement to provide optimized code according to the modification suggestions, but does not modify the function declarations in the code;

[0036] Output format: The command setting requires the reply to be made in accordance with the preset output format and does not reply to content outside the output format. If there is no code in the given content in the preset output format, it will be replaced by N / A value.

[0037] The data specific prompt generation instruction meets the following requirements:

[0038] Clarity: The data-specific prompts generated by the instruction setting requirements are concise and clear;

[0039] Specificity: The instructions set require that the generated data-specific prompts match the responses of the provided sample, that is, the generated data-specific prompts can generally describe the characteristics of the specific sample;

[0040] Analysis: Instructions set requirements for analyzing programming languages, algorithm concepts, and code characteristics, and reflecting the analysis results in data-specific prompts;

[0041] Output format: The command setting requires the reply to be in the preset output format, and only the content of the data-specific prompt is allowed to be generated.

[0042] A large language model fine-tuning training method based on multi-source code data, which constructs a fine-tuning dataset based on the multi-source code data processing method described above, and uses the fine-tuning dataset to fine-tune the large language model to be trained.

[0043] A multi-source code data processing device includes a memory, a processor, and a program stored in the memory. When the processor executes the program, the multi-source code data processing method described above is implemented.

[0044] A large language model fine-tuning training device based on multi-source code data includes a memory, a processor, and a program stored in the memory. When the processor executes the program, the large language model fine-tuning training method described above is implemented.

[0045] A storage medium stores a program thereon, which implements the multi-source code data processing method as described above when the program is executed.

[0046] A storage medium stores a program, which, when executed, implements the large language model fine-tuning training method as described above.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] (1) This paper integrates multiple existing open source code datasets and uses instruction evolution technology to enrich data diversity. It also designs data-specific prompts in combination with the relabeling idea to reconcile the inherent conflicts of multi-source code data in a fine-grained manner and improve the instruction / response consistency of the data.

[0049] (2) The present invention performs data filtering and deduplication on the reconciled multi-source code data, removes low-quality data, and improves the overall data quality.

[0050] (3) The present invention proposes to incorporate the code data construction process that reflects higher-level capabilities into the fine-tuning data, construct code understanding task data extracted from the data construction process, including instruction evolution tasks, data filtering tasks, and code review tasks, and incorporate up to multiple source code data, thereby increasing the diversity of code fine-tuning data and improving the model's code understanding ability with more dimensional data, so that the fine-tuned large language model obtains better performance and generalization capabilities.

[0051] (4) The present invention uses the harmonized multi-source code data to fine-tune the open source pre-trained large language model to obtain a fine-tuned model with powerful coding capabilities and generalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A flow chart of a method for processing multi-source code data according to the present invention;

[0053] Figure 2 generating an instruction example graph for instruction evolution data in one embodiment;

[0054] Figure 3 This is an example diagram of multiple-choice question format data constructed in one embodiment;

[0055] Figure 4 An example diagram of instructions for generating data for a code review task in one embodiment;

[0056] Figure 5 An example diagram of instructions for generating data-specific prompts in one embodiment;

[0057] Figure 6 An example diagram of a data-specific prompt generated in one embodiment;

[0058] Figure 7 A schematic diagram of the composition of a fine-tuning dataset generated in one embodiment. DETAILED DESCRIPTION

[0059] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0060] This embodiment provides a multi-source code data processing method. Figure 1 As shown, the following steps are included:

[0061] S1, open source data evolution and integration: obtain the first open source code dataset CodeAlpaca (see "Codealpaca: An instruction-following llama model for code generation") and use it as the basic data. Use gpt-turbo-3.5 as the generation model to perform instruction evolution (Instruction Evolution) on the basic data to obtain the expanded code data, that is, the instruction-evolved CodeAlpaca, recorded as EvolCode. Integrate the expanded code data EvolCode with the second open source code dataset to obtain multi-source code data, where the multi-source code data includes instructions and replies.

[0062] In this embodiment, the second open source code dataset specifically includes Evol-Instruct-Code-80k-v1 (see: https: / / huggingface.co / datasets / codefuse-ai / Evol-instruction-66k), CodeExercise-Python (see: https: / / huggingface.co / datasets / codefuse-ai / CodeExercise-Python-27k), and evol-codealpaca-v1 (see: https: / / huggingface.co / datasets / theblackcat102 / evol-codealpaca-v1).

[0063] In one embodiment, the instruction evolution data generation instruction for performing instruction evolution is as follows: Figure 2 shown.

[0064] S2, data filtering and deduplication: Filter and deduplication the data that does not meet the requirements in the multi-source code data to obtain filtered multi-source code data.

[0065] Specifically, filter out data that meets any of the following conditions:

[0066] a) The length of the responses in the multi-source data was less than 20 English words and did not contain codes;

[0067] b) The response in multiple source code data contains code that fails to compile or fails the test case (applicable to samples containing test cases in the instructions);

[0068] c) The similarity between the multi-source code data and samples in a predetermined sample set is greater than a preset threshold, that is, the N-gram similarity between the multi-source code data and samples in HumanEva (see "Evaluating large language models trained on code") and MBPP (see "Program synthesis with large language models"), the cosine similarity of the code segment embeddings, or the similarity of the code segment syntax tree is greater than 0.8.

[0069] Among them, filtering out the data that meets conditions a) and b) is filtering, and filtering out the data that meets condition c) is deduplication.

[0070] S3, code comprehension task construction: extract code comprehension task data according to the data construction process, which includes instruction evolution and data filtering and deduplication process.

[0071] The code comprehension task data in this embodiment includes:

[0072] 1) Instruction evolution task data

[0073] Specifically, data generated during the open source data evolution and integration process that meets any of the following conditions is selected as instruction evolution task data:

[0074] i) data in which the instructions before and after instruction evolution are both present in the filtered multi-source code data;

[0075] ii) the number of syntax tree levels of the reply code segment corresponding to the instruction after the instruction evolution is not less than the number of syntax tree levels of the reply code segment corresponding to the instruction before the evolution;

[0076] 2) Data filtering task data

[0077] Specifically, low-quality data is selected to construct multiple-choice question format data as data filtering task data. In this embodiment, low-quality data includes:

[0078] i) Data whose responses are too brief (response length is less than 20 English words) and do not contain codes;

[0079] ii) Data where code compilation failed;

[0080] iii) data in which the frequency of single-letter variables in the code is greater than a preset threshold and does not contain annotations;

[0081] iiii) The code is not organized into data in the functional form required by the instructions.

[0082] In one embodiment, the data filtering task data in the form of multiple choice questions is constructed as follows: Figure 3 shown.

[0083] 3) Code review task data

[0084] Specifically, a code review task data generation instruction is designed, and the code review task data is generated using multi-source code data as the basic data and gpt-4 as the generation model.

[0085] The code review task data generation instructions meet the following requirements:

[0086] 1. Professionalism: The instruction sets play the role of a professional code reviewer to conduct professional and accurate review of the given code content;

[0087] 2. Review content: The review content of the instruction settings includes correctness and clarity. Correctness means that the code has no syntactic and logical errors and implements the given function name. Clarity means that variables should have meaningful names and function parameters and return values should have type annotations.

[0088] 3. Scoring criteria: The instructions set a preset score range (0-10 points), score the code based on correctness and clarity, and provide corresponding review comments;

[0089] 4. Suggestions for modification: The directive sets requirements for providing suggestions for further improvement;

[0090] 5. Optimize code: The instruction setting requires providing optimized code according to the modification suggestions, but does not modify the function declarations appearing in the code;

[0091] 6. Output format: The command setting requires that the reply be made according to the preset output format and no reply be made to content outside the output format. If there is no code in the given content in the preset output format, it will be replaced with N / A value.

[0092] In one embodiment, the code review task data generation instruction is as follows: Figure 4 shown.

[0093] S4, data-specific prompt generation: Design data-specific prompt generation instructions, use the filtered multi-source code data as the basic data, use gpt-4 as the generation model to generate data-specific prompts, and insert the generated prompts into the beginning of the corresponding original basic data instructions to form data-specific prompt customized data.

[0094] Data-specific hint generation instructions meet the following requirements:

[0095] 1. Clarity: The instruction setting requires that the data-specific prompts generated be concise and clear to make the prompts clearer;

[0096] 2. Specificity: The instructions set require that the generated data-specific prompts match the responses of the provided sample, that is, the generated data-specific prompts can generally describe the characteristics of the specific sample;

[0097] 3. Analysis: The instruction set requires analysis of programming language, algorithm concepts and code characteristics, and briefly reflects the analysis results in data-specific prompts;

[0098] 4. Output format: The command setting requires the reply to be made in accordance with the preset output format, and only the content of the data-specific prompt is allowed to be generated. Other parts cannot be modified.

[0099] The purpose of generating data-specific prompts is to improve the quality of fine-tuning data. This is specifically reflected in two aspects. First, data-specific prompts generated from the same model have similar language style and quality, which can bridge the domain differences between data from different sources. Second, data-specific prompts are highly correlated with the data's responses, which can better align the data's instructions and responses, making the large model fine-tuned on this data have stronger instruction-following capabilities.

[0100] In one embodiment, the data specific prompt generation instruction is as follows: Figure 5 As shown, the generated data-specific prompts are as follows Figure 6 shown.

[0101] S5, data merging: Merge the code comprehension task data and data-specific prompt customization data into the filtered multi-source code data to achieve data reconciliation.

[0102] This embodiment also provides a large language model fine-tuning training method based on multi-source code data, which constructs a fine-tuning dataset based on the multi-source code data processing method described above, and uses the fine-tuning dataset to fine-tune the large language model to be trained.

[0103] In one embodiment, after the above-mentioned coordinated multi-source code data processing flow, the fine-tuning data set composition (outer loop) and programming language distribution (inner loop) are formed as follows: Figure 7As shown in the figure, the fine-tuning dataset includes six types of data: three types of existing code data: Evol-Instruct-Code-80k-v1, CodeExercise-Python, and evol-codealpaca-v1; three types of data generated through multi-source data processing: CodeAlpaca (EvolCode in the outer ring of the figure), code comprehension task data, and customized data with specific prompts, totaling 24.3% + 3.7% + 4.6%. The inner ring summarizes the programming language distribution of the code included in the fine-tuning data. When fine-tuning a large model, the language of choice in the code fine-tuning data depends on specific needs. For example, to achieve more general multi-language code capabilities, the fine-tuning data should include code from different programming languages. (Due to the popularity and ease of use of Python, the proportion of data containing Python code will be relatively high. Similarly, the proportion of data containing less popular programming languages will be lower.) To achieve a large model with stronger Python code capabilities, the proportion of data containing Python code should be further increased, or even exclusively included. Thanks to the integration of multi-source data, the fine-tuning dataset of the present invention includes codes of various mainstream programming languages. The large model fine-tuned from this data can obtain powerful general multi-programming language code capabilities.

[0104] Use as Figure 7 We used the fine-tuning dataset and CodeLlama-Python-7B (see Code llama: Openfoundation models for code) as the base language model for two rounds of fine-tuning, resulting in a model with better code capabilities and generalization performance. The performance comparison is shown in Table 1.

[0105] Table 1

[0106]

[0107] According to Table 1, the present invention improves the average performance by 23.8 on the base model, significantly surpassing the current large language model with the same parameter scale.

[0108] In addition, it should be noted that data generation incurs certain costs, including model inference costs (OpenAI API monetary costs) and time costs. However, the code comprehension task data and data-specific prompt customization data, totaling 3.7% + 4.6%, are more critical. They achieve significant performance improvements with a small amount of generated data, that is, a relatively small cost. According to Table 1, the average index improvement is 12.5.

[0109] The present invention is not limited to a specific type of single code dataset. It introduces code understanding tasks and data-specific prompts, has better data diversity, and reconciles the inherent conflicts of multiple source code data. It can better stimulate the capabilities of the base large language model through fine-tuning.

[0110] Moreover, the fine-tuning method of the present invention enables the fine-tuned large language model to demonstrate better performance and stronger generalization capabilities in multiple mainstream tasks, including Python code generation, mathematical science code generation, natural language understanding, and mathematical reasoning. This is because the data-specific prompts generated by the present invention bridge the gap between commands / responses in the data, enabling the large language model fine-tuned on this data to have better command-following capabilities.

[0111] This embodiment further provides a multi-source code data processing device, comprising a memory, a processor, and a program stored in the memory, wherein the processor implements the multi-source code data processing method described above when executing the program.

[0112] This embodiment also provides a large language model fine-tuning training device based on multi-source code data, including a memory, a processor, and a program stored in the memory. When the processor executes the program, it implements the large language model fine-tuning training method described above.

[0113] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0114] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A multi-source code data processing method, characterized in that: The following steps are involved: Open source data evolution and integration: obtaining a first open source code dataset and using it as base data, performing instruction evolution on the base data using the first language model as a generative model to obtain expanded code data, and integrating the expanded code data with a second open source code dataset to obtain multi-source code data, wherein the multi-source code data includes instructions and responses; Data filtering and deduplication: Filter and dedupe the data that does not meet the requirements in the multi-source code data to obtain the filtered multi-source code data; Code comprehension task construction: Extract code comprehension task data according to the data construction process, which includes instruction evolution and data filtering and deduplication process; The code comprehension task data includes: 1) Instruction evolution task data: Data generated during the evolution and integration of open source data that meets any of the following conditions is selected as instruction evolution task data: The instructions before and after instruction evolution are both present in the filtered multi-source code data; The number of syntax tree levels of the reply code segment corresponding to the instruction after the instruction evolution is not less than the number of syntax tree levels of the reply code segment corresponding to the instruction before the evolution; 2) Data filtering task data: Select low-quality data to construct multiple-choice format data as data filtering task data; 3) Code review task data: Design instructions for generating code review task data. Use multiple source code data as the base data and the second largest language model as the generation model to generate code review task data. Data-specific prompt generation: Design data-specific prompt generation instructions, use the filtered multi-source code data as the basic data, use the second largest language model as the generation model to generate data-specific prompts, and insert the generated prompts into the beginning of the corresponding original basic data instructions to form data-specific prompt customized data; Data merging: Merge code comprehension task data and data-specific prompt customized data into the filtered multi-source code data to achieve data reconciliation.

2. A multi-source code data processing method according to claim 1, characterized in that: The data filtering and deduplication specifically involves filtering out data that meets any of the following conditions: a) The length of the response in the multi-source code data is less than the preset threshold and does not contain the code; b) Multiple source code data responses contain code that fails to compile or pass test cases; c) The similarity between the multi-source code data and the samples in the predetermined sample set is greater than a preset threshold.

3. A multi-source code data processing method according to claim 2, characterized in that: The similarity between the multiple source code data and samples in the predetermined sample set is greater than a preset threshold specifically means that any one of the N-gram similarity, the cosine similarity of the code segment embedding, or the similarity of the code segment syntax tree between the multiple source code data and the samples in the predetermined sample set exceeds the preset threshold.

4. A multi-source code data processing method according to claim 1, characterized in that: The low-quality data include: The length of the response is less than the preset threshold and does not contain data with a code; Data on code compilation failures; The frequency of single-letter variables in the code is greater than the preset threshold and does not contain comments; Failure to organize the code into data in the functional form required by the instructions; The code review task data generation instruction meets the following requirements: Professionalism: The instruction sets act as professional code reviewers to review the given code content; Review content: The review content of the directive settings includes correctness and clarity. Correctness means that the code has no syntactic and logical errors and implements the given function names. Clarity means that variables should have meaningful names and function parameters and return values should have type annotations. Scoring criteria: Instructions set a preset score range, score the code based on correctness and clarity, and provide review comments; Suggestions for modification: The directive sets requirements to provide suggestions for further improvement; Optimize code: The instruction sets the requirement to provide optimized code according to the modification suggestions, but does not modify the function declarations in the code; Output format: The command setting requires the reply to be made in accordance with the preset output format and does not reply to content outside the output format. If there is no code in the given content in the preset output format, it will be replaced by N / A value.

5. The multi-source code data processing method according to claim 1, characterized in that: The data specific prompt generation instruction meets the following requirements: Clarity: The data-specific prompts generated by the instruction setting requirements are concise and clear; Specificity: The instructions set require that the generated data-specific prompts match the responses of the provided sample, that is, the generated data-specific prompts can generally describe the characteristics of the specific sample; Analysis: Instructions set requirements for analyzing programming languages, algorithm concepts, and code characteristics, and reflecting the analysis results in data-specific prompts; Output format: The command setting requires the reply to be in the preset output format, and only the content of the data-specific prompt is allowed to be generated.

6. A large language model fine-tuning training method based on multi-source code data, characterized in that: A fine-tuning dataset is constructed based on the multi-source code data processing method according to any one of claims 1 to 5, and the fine-tuning dataset is used to perform fine-tuning training on the large language model to be trained.

7. A multi-source code data processing device comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the multi-source code data processing method according to any one of claims 1 to 5 is implemented.

8. A large language model fine-tuning training device based on multi-source code data, comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the large language model fine-tuning training method according to claim 6 is implemented.

9. A storage medium having a program stored thereon, characterized in that: When the program is executed, the multi-source code data processing method according to any one of claims 1 to 5 is implemented.

10. A storage medium having a program stored thereon, characterized in that: When the program is executed, the large language model fine-tuning training method as claimed in claim 6 is implemented.

Images