Methods, computer programs, and computer systems (decontamination of training data for code generation models)

Abstract

This invention provides a method, program, and system for performing decontamination of training data for a training code generation model. [Solution] The method performs content matching between benchmark data and training data to identify content-shared data containing strings common to both, identifies one or more repositories containing instances of the content-shared data, generates one or more contamination metrics for each repository based on the frequency of instances of the content-shared data in one or more repositories, identifies one or more contaminated repositories based on the contamination metrics for each repository, decontaminates the training data based on the identification of one or more contaminated repositories and metadata of training data samples in the training data, and trains a machine learning computer model based on the decontaminated training data.

Description

Technical Field

[0001] This application generally relates to data processing apparatuses and methods, and more specifically, to computing tools for performing contamination removal of training data for a training code generation model and the operations / functions of the computing tools.

Background Art

[0002] Large language models (LLMs) such as ChatGPT are advanced machine learning computer models designed to understand and generate human language. LLMs are trained on large amounts of text data, such as documents available via the Internet, various curated document repositories, and the like. Thereby, LLMs are capable of performing a wide range of natural language processing (NLP) tasks such as text generation, translation, summarization, and question answering.

[0003] In recent years, some LLMs have been trained to perform computer code generation tasks. That is, a code generation LLM receives a prompt or instruction for generating a portion of code for performing a specified operation, and the LLM returns a portion of the code that performs that operation. Such code generation LLMs assist developers in writing computer code, finding errors, and translating between programming languages.

Summary of the Invention

Problems to be Solved by the Invention

[0004] Code generation LLMs may not operate correctly in code generation scenarios that have not been pre-checked, but may be inaccurately evaluated as having better performance during evaluation based on benchmark data.

Means for Solving the Problems

[0005] This summary of the invention is provided in a simplified form to introduce a selection of concepts that will be further described in the embodiments for carrying out the invention herein. This summary of the invention is not intended to identify any important factors or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0006] In one exemplary embodiment, a method is provided comprising the step of performing content matching between benchmark data and training data, thereby identifying content-shared data, wherein the content-shared data includes a string common to the benchmark data and the training data. The method further comprises the step of identifying one or more repositories containing instances of the content-shared data, and the step of generating one or more contamination metrics for each repository in the one or more repositories based on the frequency of the instances of the content-shared data in the one or more repositories. The method also comprises the step of identifying one or more contaminated repositories in the one or more repositories based on the one or more contamination metrics for each repository. In addition, the method comprises the step of decontaminating the training data based on the identification of the one or more contaminated repositories and metadata of training data samples in the training data. The method further comprises the step of training a machine learning computer model based on the decontaminated training data.

[0007] In other exemplary embodiments, a computer program product is provided which comprises a computer-readable program on a computer-readable medium. When the computer-readable program is executed on a computing device, the computing device is caused to perform various operations and combinations thereof from among the operations outlined above with respect to exemplary embodiments of the method.

[0008] In yet another exemplary embodiment, a system / device is provided. The system / device may comprise one or more processors and memory coupled to the one or more processors. The memory may, when executed by the one or more processors, have instructions that cause the one or more processors to perform various operations and combinations thereof from the operations outlined above with respect to exemplary embodiments of the method.

[0009] These and other features and advantages of the present invention will be described in the following detailed description of exemplary embodiments of the present invention, or will become apparent to those skilled in the art in consideration of such detailed description. [Brief explanation of the drawing]

[0010] The present invention, as well as preferred modes of use and further objectives and advantages, will be best understood by referring to the following detailed description of exemplary embodiments in conjunction with the accompanying drawings.

[0011] [Figure 1] This figure shows an example of a distributed data processing system environment in which exemplary embodiments can be implemented and at least some of the computer code involved in carrying out the method of the present invention can be executed.

[0012] [Figure 2] This is an example block diagram illustrating the key operating components of a decontamination system for a code generation LLM training dataset, according to one exemplary embodiment.

[0013] [Figure 3A] This is a diagram illustrating one example of a benchmark data sample from one exemplary embodiment.

[0014] [Figure 3B] This is a diagram illustrating an example of a repository record having corresponding metadata, according to one exemplary embodiment.

[0015] [Figure 4] This is a flowchart illustrating the operation of an example of a decontamination system according to one exemplary embodiment. [Modes for carrying out the invention]

[0016] Exemplary embodiments provide computing tools and the behavior / functions of computing tools for performing decontamination of training data for training code-generating models, such as large-scale code-generating language models (LLMs). Exemplary embodiments provide computing tools that identify repositories of training data that may be contaminated with benchmark data and use metadata of the training data to determine which portions of the training data may originate from such repositories. Exemplary embodiments then decontaminate the training data by removing potentially contaminated training data, thereby removing benchmark data. The decontaminated training data may then be used as the basis for training a code-generating LLM.

[0017] In particular, a problem that the present invention particularly aims to solve when training a code-generating LLM is the inclusion of benchmark data in the training dataset used to train the code-generating LLM. The benchmark data is used to evaluate the code-generating LLM and ensure that it functions adequately with respect to generating code based on a prompt or instruction. With respect to a code-generating LLM, the benchmark data may include a sample or instance of an instruction or prompt that describes computer code to perform a specified operation or function, along with the corresponding portion of the computer code (or simply "code" as used herein) that the LLM should generate to perform the function or operation, i.e., the normative answer. In some cases, the benchmark may include a portion of source code as a prompt, and the answer may be the next line of code or a code snippet. The benchmark data may be created manually or obtained from a code repository, such as a Git repository. Using such a sample of benchmark data, a trained code-generating LLM may be tested to determine how close the output of the trained code-generating LLM based on a benchmark instruction / prompt is to the benchmark code.

[0018] Various benchmark datasets exist for evaluating such code generation LLMs, including MBPP, CanltE-dit, DS1000, HumanEval, CodeNet, CrossCodeEval, and similar datasets. However, when benchmark data is present within the training data, the code generation LLM is trained on the benchmark data as well as the other training data. This means the code generation LLM is specifically trained on the benchmark case, potentially biasing the evaluation of the code generation LLM to show higher performance than it would otherwise. As a result, the code generation LLM may not function correctly in untested code generation scenarios, but may be inaccurately evaluated as performing better during evaluations based on benchmark data.

[0019] Therefore, it is crucial to decontaminate the training data used to train code-generating LLMs before training them, for example, by removing instances of benchmark data from the training data. In natural language versions of LLMs, decontamination of training datasets can be performed by performing semantic similarity evaluations. However, due to significant differences between benchmark datasets and code-generating tasks, such mechanisms do not work for code-generating LLMs. That is, for code-generating LLMs, benchmark datasets are diverse and can include code-generating benchmark data, code-translation benchmark data, mathematical problems, code descriptions, and similar. In code-generating tasks, strict syntax is important, and unlike question-answer pairs in language model benchmarks, semantic similarity is not a good measure for detecting potential contamination in code-generating LLMs. This is especially true because code-generating benchmark datasets often contain short code snippets, such as "return x+y," and using semantic similarity to decontaminate training data that may have similar but different code snippets can produce many false positives. The decontamination process itself can potentially degrade the code generation LLM. For example, if the benchmark asks you to "write code for the Fibonacci sequence," should you remove all documents containing the term "Fibonacci" from the training data?

[0020] Therefore, the decontamination problem of training datasets with benchmark data includes false negatives, in which case problematic data remains in the training dataset undetected, even if there is overlap between the benchmark and training sets, and this overlap may make it easier for the LLM to solve the test cases. Detection failures can be caused by differences in format and representation between the benchmark and training datasets. Furthermore, the problem includes false positives, in which harmless code is mistakenly considered contaminated due to the use of short, generic code snippets in the benchmark and training datasets. Overdetection of contamination can reduce or skew the training data, potentially degrading the quality of the trained code-generating LLM. In addition, the problem of decontamination of training datasets includes the computational cost of decontamination, for example, because the cost of matching all pairs of benchmark and training sets cannot be ignored due to the large size of the training data.

[0021] The exemplary embodiments provide computing tools and their behavior / functions that specifically target the decontamination of training data for training code-generating LLMs. While the primary exemplary embodiments are described in the context of training code-generating LLMs, it should be understood that the mechanisms of the exemplary embodiments may be implemented with respect to training datasets for other types of machine learning computer models where the presence of benchmark data within the training dataset can cause problems in training the machine learning computer model. For this reason, the exemplary embodiments may also be implemented to improve the training of such other types of machine learning computer models and are not limited to code-generating LLMs. Embodiments of code-generating LLMs are provided herein as examples, and those skilled in the art will readily identify other applications of this exemplary embodiment to other types of machine learning computer models in light of this description.

[0022] Exemplary embodiments perform content matching between benchmark data and training data to identify instances of benchmark data within the training data. That is, string matching is performed between a training data set and a benchmark data set. If a string within the benchmark data set exists within the training data set, these common strings, referred to herein as content sharing data, are considered potential contamination, and the corresponding benchmark data set ID is attached to the training data as an annotation so that it can be removed from the training data set. Before string matching, normalization such as removal of whitespace, removal of special characters (e.g., newline characters), conversion from uppercase to lowercase, and the like is applied. Additionally, other techniques including exclusion of short strings having a length below a given threshold may be used to enhance string matching, thereby excluding frequently used code snippets that can result in false positives. Comment removal may be used to remove comment lines within the code before performing string matching.

[0023] To reduce the computational cost of string matching, if it can be determined using the metadata of the training data that string matching can be skipped, for example, if the programming languages of the benchmark data and the training data are different. Additionally, the training data may be cleansed by performing duplicate elimination on the training data samples. Moreover, string duplicate elimination may be performed such that if string duplicates exist within the same benchmark data set, they are integrated into one string for the purpose of string matching with the training data set.

[0024] Content sharing data is used to search for instances of content sharing data within a repository to determine which repositories are potentially contaminated with benchmark data. A repository stores documents that are used by a code generation LLM to perform code generation operations. The documents may have various types of content including actual computer code in various programming languages, code descriptions in natural language text, issues and commits of computer code, and the like. One example of a code repository is starcoderdata, a repository of training data used to train the StarCoder language model for code generation.

[0025] Repositories may be scored and ranked based on the frequency of occurrence of content sharing data. For example, the scoring may be a function of an evaluation of the number of occurrences of content sharing data within a repository, the number of unique instances of content sharing data, the proportion of the repository containing content sharing data relative to the total size of the repository, or the like. Based on the scoring and / or ranking of the frequency of occurrence of content sharing data, a list of repositories and scores / ranks indicating which repositories are more / less contaminated with benchmark data may be generated. A threshold may be established to determine when the degree of potential contamination is such that a repository should be considered contaminated for the purpose of cleaning the training dataset, which will be described later.

[0026] In some exemplary embodiments, it is possible (and simplest) to determine that a repository is contaminated if it contains at least one instance (document) of content-sharing data, but more complex and diverse embodiments may also be implemented. For example, suppose there are two repositories, A and B. Repository A contains 100 documents, and repository B contains 10,000 documents, and both repositories A and B contain 5 instances of content-sharing data. Then, for repository X, r(X), i.e., the degree of contamination of X, can be defined as the ratio of the number of documents of content-sharing data in X to the total number of documents in X. In this case, r(A) = 0.05 and r(B) = 0.0005, which indicates that repository A is more likely to be contaminated than repository B. If the threshold t is set to, for example, 0.001 for r, then r(A) > 0.001 holds, but r(B) > 0.001 does not. In an exemplary embodiment, this inequality can be used to determine whether a document in X should be removed from the training data by checking whether r(X) > t holds. A desirable property of the function r is that the value of r is between 0 and 1, and that a larger value of r(X) indicates a higher probability that X is contaminated. Any function r that satisfies this property is available for scoring.

[0027] In some exemplary embodiments, after assigning a score to repository-based matching as discussed above, the repository score is combined with a content-based matching score, which may be a similarity score. For example, this similarity score may be based on syntactic and semantic similarity (0 ≤ similarity ≤ 100, which can be normalized as 0 ≤ c ≤ 1). Both the content score and the repository score may be used for decontamination decisions (whether to exclude a document from the training data), so that the total score s may be defined as s = (1 - α)c + αr, where c is the content matching score, r is the degree of contamination of the source repository, α is an empirically set value, and a threshold θ is also empirically determined, where the data is decontaminated if s > θ. For example, by empirically setting α=0.9 and θ=0.1, a document in the training dataset with a similarity score c=0.8 and r=0.1 with a specific benchmark instance (e.g., 10 out of 100 data points in the repository are content-shared data) would have a similarity score of s=0.1*0.8+0.9*0.1=0.17, which exceeds the threshold θ, and therefore the document is removed from the training dataset.

[0028] The concise case of a repository having the aforementioned arbitrary content-sharing data can also be formalized as follows: c=1 if the string of test data is a substring of the training data, otherwise 0; r=1 if the repository has 50 or more content-sharing data, otherwise 0; α=0.5; and θ=0.1. Potential contamination can then be found as either string matching or any dataset from the contaminated repository. These are merely examples of some exemplary embodiments of scoring with matching and benchmarks, and scoring of repositories, and are not intended to limit possible embodiments. Any other suitable scoring methods may be used without departing from the spirit and scope of the invention, as long as they achieve the objective of identifying repositories that may be potentially contaminated.

[0029] A list of potentially contaminated repositories—that is, repositories containing content-sharing data to a degree sufficient to be considered contaminated for the purpose of decontamination of the training dataset—may be used to identify which portions of the training data are actual contaminated by benchmark data and which portions may be false positives. That is, many benchmark datasets are generated from such repositories, and therefore repositories may contain benchmark data. For this reason, an exemplary embodiment identifies which repositories have instances of content-sharing data, i.e., data that has a common string between the benchmark data and the training data, and then determines which portions of the content-sharing data in the training data originate from a repository that has been determined to be contaminated. These represent true positives of potential contamination, while other instances that may not originate from a contaminated repository may be considered false positives.

[0030] Furthermore, in some exemplary embodiments, the exemplary embodiments may be used to reduce instances of false negatives in content matching by assuming that all training data originating from a contaminated repository is indeed contaminated within the training dataset. This may also apply to forked repositories originating from a contaminated repository, since forked repositories can also be assumed to be contaminated. In such embodiments, the training dataset may be decontaminated not only by removing instances of content-sharing data detected based on string matching between the benchmark data and the training dataset, but also based on metadata specifying the repository source of the training data. That is, for training data samples originating from a contaminated repository, those training data samples may be removed from the training dataset without the need to perform string matching between the benchmark data and the training dataset. Then, the remaining training dataset samples may be subjected to string matching between the benchmark dataset and the training dataset to identify instances of content-sharing data, which may then be removed from the training dataset.

[0031] Therefore, exemplary embodiments provide computer tools and computer operations / functions for decontaminating the training dataset with respect to benchmark data. This may be done using a combination of metadata-based identification and string matching of training data samples originating from identified contaminated repositories. In some exemplary embodiments, false positives and true positives may be distinguished based on such metadata and identification of contaminated repositories. The resulting decontaminated training dataset may then be provided to the training logic for training a code-generating LLM. As a result, the trained code-generating LLM is not trained on benchmark data, and therefore the benchmark dataset can be accurately used to evaluate the performance of the trained code-generating LLM.

[0032] The following description provides examples of embodiments of the present disclosure, and modifications and substitutions may be made in other embodiments. Herein, several examples are provided to further clarify the various aspects of the present disclosure.

[0033] Example 1: A method comprising the steps of performing content matching between benchmark data and training data, thereby identifying content-shared data, wherein the content-shared data includes strings common to the benchmark data and the training data. The method further comprises the steps of identifying one or more repositories containing instances of the content-shared data, and generating one or more contamination metrics for each repository in the one or more repositories based on the frequency of the instances of the content-shared data in the one or more repositories. The method also comprises the steps of identifying one or more contaminated repositories in the one or more repositories based on the one or more contamination metrics for each repository. In addition, the method comprises the steps of decontaminating the training data based on the identification of the one or more contaminated repositories and metadata of training data samples in the training data. Furthermore, the method comprises the steps of training a machine learning computer model based on the decontaminated training data. The above limitations favorably enable improved training of the machine learning computer model by efficiently removing benchmark data from the training data based on the identification of contaminated repositories from which the training data can be obtained.

[0034] Example 2: A limitation of any of Examples 1 and 3-10, wherein the step of decontaminating the training data comprises: for each training data sample in the training data, comparing one or more first attributes of the training data sample in the metadata of the training data sample with one or more second attributes of the one or more contaminated repositories; and removing the training data sample from the training data in response to identifying a match between at least one first attribute of the training data sample and at least one second attribute of a contaminated repository among the one or more contaminated repositories. The above limitation favorably enables the removal of training data samples having attributes that match attributes of contaminated repositories in order to ensure that the training data samples of the training data do not have potential contamination by benchmark data.

[0035] Example 3: The limitation described in any of Examples 1-2 and 4-10, wherein the one or more first attributes include an identifier of the source repository from which the corresponding training data sample was obtained, and the one or more second attributes include a repository identifier of the corresponding contaminated repository. The limitation described above favorably allows for the removal of training data samples originating from contaminated repositories in order to ensure that the training data is free from contamination from these contaminated repositories.

[0036] Example 4: The limitation of any of Examples 1-3 and 5-10, wherein the step of decontaminating the training data includes a step of performing preprocessing on the training data to identify portions of the training data that are not subject to decontamination, and the preprocessing includes identifying portions of the training data in a computer programming language different from the programming language of the benchmark data. The limitation above favorably enables a more efficient evaluation of content sharing data between the benchmark data and the training data by skipping portions in different programming languages, and therefore portions that are not likely to have benchmark data from benchmark datasets in different programming languages.

[0037] Example 5: The limitation described in any of Examples 1-4 and 6-10, wherein the one or more contamination metrics include at least one of the following: the number of instances of content-sharing data in the one or more repositories, or the number of unique instances of content-sharing data in the one or more repositories. The limitation described above favorably enables the distinction between contaminated and uncontaminated repositories based on an assessment of the frequency of occurrence of content-sharing data in the repositories.

[0038] Example 6: The limitation described in any of Examples 1-5 and 7-10, wherein the one or more contamination metrics include the degree of contamination of the corresponding repository, the degree of contamination being a function of the ratio of the number of documents in the corresponding repository having instances of the content-sharing data to the total number of documents in the corresponding repository. The limitation described above favorably enables the distinction between contaminated and uncontaminated repositories based on an assessment of the degree of contamination of the repository.

[0039] Example 7: The step of performing content matching between benchmark data and training data to identify the content-sharing data further comprises the step of annotating training data samples in the training data that have the content-sharing data with benchmark annotations specifying identifiers of the benchmark data, wherein the training data samples annotated with benchmark annotations are candidates for removal from the training data, as described in any of Examples 1-6 and 8-10. The above limitation favorably enables marking of training data samples that are candidates for removal from the training data, thereby allowing these marked training data samples to be further evaluated to determine whether they should be removed from the training data.

[0040] Example 8: The one or more contamination metrics described above include a function of the length of the content-sharing data string instances present in the corresponding repository, as defined in any of Examples 1-7 and 9-10. The above limitation favorably enables the identification of contaminated repositories based on the length of the content-sharing data string instances in the repository, recognizing that shorter lengths are less likely to represent contamination and longer lengths are more likely to represent contamination.

[0041] Example 9: The machine learning computer model is a code-generating large-scale language model (LLM), as defined in any of Examples 1-8 and 10. The above limitation favorably enables improved training of the code-generating LLM by minimizing or eliminating benchmark data in the training data, thereby allowing the benchmark data to be accurately used to evaluate the performance of the trained code-generating LLM.

[0042] Example 10: The limitation in any of Examples 1-9, wherein the one or more repositories are code generation repositories containing documents used by the code generation LLM to generate computer code for performing a function specified in a prompt input to the code generation LLM. The limitation described above favorably enables the identification of code generation repositories that may be impaired by including benchmark data, and by including benchmark data, the trained code generation LLM may not be able to properly evaluate using the benchmark data.

[0043] Example 11: A system comprising one or more processors and one or more computer-readable storage media collectively storing program instructions, wherein the program instructions, when executed by the one or more processors, are configured to cause the one or more processors to execute the method described in any one of Examples 1 to 10. The above limitation is advantageous in enabling a system comprising one or more processors to perform and realize the advantages described with respect to Examples 1 to 10.

[0044] Example 12: A computer program product comprising one or more computer-readable storage media and program instructions collectively stored in the one or more computer-readable storage media, wherein the program instructions include instructions configured to cause the one or more processors to perform the method described in any one of Examples 1 to 10. The above limitations favorably enable computer program products having program instructions configured to cause one or more processors to perform and realize the advantages described with respect to Examples 1 to 10.

[0045] Before continuing our discussion of exemplary embodiments and various aspects of the improved computer operations performed by such exemplary embodiments, it should first be understood that throughout this description, the term “mechanism” is used to refer to elements of the present invention that perform various operations, functions, and similar ones. Where the term is used herein, “mechanism” may be an implementation of a function or aspect of an exemplary embodiment in the form of an apparatus, procedure, or computer program product. In the case of a procedure, the procedure is implemented by one or more devices, apparatus, computer, data processing system, or similar. In the case of a computer program product, logic represented by computer code or instructions embodied within or on the computer program product is executed by one or more hardware devices to implement a function or perform an operation associated with a particular “mechanism.” Therefore, the mechanisms described herein may be implemented as special hardware, software executed on the hardware that configures the hardware to implement special functions of the present invention that the hardware would not otherwise be able to perform, software instructions stored on a medium that make the instructions easily executable by the hardware, thereby specially configuring the hardware to perform the functions and specific computer operations described herein, or any combination thereof.

[0046] This specification and the claims may use the terms “one,” “at least one of,” and “one or more of” with respect to certain features and elements of exemplary embodiments. It should be understood that these terms and phrases are intended to state that at least one of certain features or elements present in a particular exemplary embodiment is present, but more than one may be present. That is, these terms / phrases are not intended to limit this specification or the claims to a single feature / element present, nor are they intended to require the presence of multiple such features / elements. On the contrary, these terms / phrases merely require at least a single feature / element, and multiple such features / elements may be present within the scope of this specification and the claims.

[0047] Furthermore, the use of the term “engine” as used herein in connection with the description of embodiments and features of the present invention is not intended to limit any particular technical implementation for achieving and / or performing actions, stages, processes, etc. caused by and / or performed by an engine, but it should be understood that “engine” is limited in that even if it is implemented in computer technology and the engine can function in combination with manual input or provide output intended for manual or mental consumption, its actions, stages, processes, etc. are not performed as mental processes or performed through manual effort. An engine is implemented as one or more of software, dedicated hardware, and / or firmware, or any combination thereof, running on hardware, specifically configured to perform a specified function. Hardware may include, but is not limited to, the use of a processor loaded or stored in machine-readable memory and executed by a processor, thereby combining the processor with appropriate software that specifically configures the processor for a particular purpose, including one or more of the functions of one or more embodiments of the present invention. Furthermore, any names associated with a particular engine are for convenience of reference only and are not intended to limit to a particular implementation, unless otherwise specified. In addition, any function attributed to a particular engine may be performed equally by multiple engines, incorporated into and / or combined with the functions of another engine of the same or different type, or distributed across one or more engines in various configurations.

[0048] In addition, it should be understood that the following description is intended to further illustrate the exemplary implementation of the exemplary embodiment and to aid in understanding the mechanism of the exemplary embodiment by using several different examples of various elements of the exemplary embodiment. These examples are intended to be non-limiting and do not exhaust all possible ways of implementing the mechanism of the exemplary embodiment. It will be apparent to those skilled in the art, in consideration of this specification, that there are many other alternative implementations of these various elements that can be used in addition to or instead of the examples provided herein without departing from the spirit and scope of the invention.

[0049] Various aspects of this disclosure are described by descriptive text, flowcharts, block diagrams of computer systems and / or block diagrams of machine logic included in computer program product (CPP) embodiments. For any flowchart, depending on the technology involved, operations may be performed in an order different from that shown in a given flowchart. For example, again, depending on the technology involved, two operations shown in consecutive flowchart blocks may be performed in reverse order, as a single integrated step, simultaneously, or at least partially overlapping in time.

[0050] Embodiments of a computer program product ("CPP Embodiment" or "CPP") are terms used in this disclosure to describe any set of storage media (also called "mediums") that collectively comprise a set of storage devices that collectively comprise machine-readable code corresponding to instructions and / or data for performing computer operations specified in a given CPP claim. A "storage device" is any tangible device capable of holding and storing instructions for use by a computer processor. Computer-readable storage media may be, but are not limited to, electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, mechanical storage media, or any suitable combination thereof. Some known types of storage devices, including these media, include diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disk read-only memory (CD-ROM), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices (such as pits / lands formed on the main surface of punch cards or disks), or any preferred combination of those described above. When the term "computer-readable storage medium" is used in this disclosure, it shall not be construed as storage in the form of transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides, optical pulses passing through optical fiber cables, electrical signals communicated through wires, and / or other transmission media. As those skilled in the art will understand, data is moved at several intermittent points in the normal operation of a storage device, such as during access, defragmentation, or garbage collection; however, data is not transient while it is stored, so the foregoing does not make a storage device transient.

[0051] It should be understood that certain features of the invention described in the context of separate embodiments for clarity may be provided in combination in a single embodiment. Conversely, various features of the invention described in the context of a single embodiment for brevity may be provided separately or in any preferred partial combination.

[0052] The present invention may be a specially configured computing system comprising hardware and / or software specifically configured to implement certain mechanisms and functions described herein, a method implemented by the specially configured computing system, and / or a computer program product including software logic loaded into the computing system to specifically configure the computing system to implement the mechanisms and functions described herein. Whether described as a system, a method, or a computer program product, it should be understood that the exemplary embodiments described herein particularly focus on improved computing tools and methodologies implemented by such improved computing tools. In particular, the improved computing tools of the exemplary embodiments particularly provide decontamination of training datasets based on metadata specifying the source of training data and string matching. The improved computing tools implement mechanisms and functions such as a decontamination system for training data of a code generation LLM, which cannot be practically performed by a human, either outside or with the assistance of a technical environment such as mental processing or the like. The improved computing tools offer a practical application of the method, at least in that they can improve the training of code-generating LLMs by decontaminating the training dataset to remove benchmark data, thereby making the benchmark dataset available for evaluating the performance of the trained code-generating LLM.

[0053] Figure 1 is a diagram of an example of a distributed data processing system environment in which an exemplary embodiment can be implemented and in which at least some of the computer code involved in performing the method of the present invention can be executed. That is, the computing environment 100 includes an example of an environment for executing at least some of the computer code involved in performing the method of the present invention, such as a decontamination system 200 for training data of a code generation LLM, which is referred to herein simply as the decontamination system 200. In addition to the decontamination system 200, the computing environment 100 includes, for example, a computer 101, a wide area network (WAN) 102, an end-user device (EUD) 103, a remote server 104, a public cloud 105, and a private cloud 106. In this embodiment, the computer 101 includes a processor set 110 (including processing circuits 120 and cache 121), a communication fabric 111, volatile memory 112, persistent storage 113 (including an operating system 122 and the decontamination system 200 identified above), a peripheral device set 114 (including a user interface (UI), a device set 123, storage 124, and an Internet of Things (IoT) sensor set 125), and a network module 115. The remote server 104 includes a remote database 130. The public cloud 105 includes a gateway 140, a cloud orchestration module 141, a host physical machine set 142, a virtual machine set 143, and a container set 144.

[0054] Computer 101 can take the form of a desktop computer, laptop computer, tablet computer, smartphone, smartwatch or other wearable computer, mainframe computer, quantum computer, or any other form of computer or mobile device currently known or to be developed in the future that is capable of running programs, accessing networks, or querying databases such as remote database 130. As is well understood in the field of computer technology, and depending on the technology, the execution of a computer implementation method can be distributed among multiple computers and / or multiple locations. On the other hand, in this presentation concerning the computing environment 100, in order to make the presentation as concise as possible, the detailed considerations focus on a single computer, specifically computer 101. Although computer 101 is not shown in the cloud in Figure 1, it may be located in the cloud. On the other hand, computer 101 does not need to be located in the cloud, except to any extent that can be definitively shown.

[0055] The processor set 110 includes one or more computer processors of any type currently known or to be developed in the future. The processing circuitry 120 may be distributed across multiple packages, for example, multiple, coordinated integrated circuit chips. The processing circuitry 120 may implement multiple processor threads and / or multiple processor cores. The cache 121 is memory located in the processor chip package and is typically used for data or code that should be available for rapid access by threads or cores running on the processor set 110. The cache memory is typically organized into multiple levels depending on its relative proximity to the processing circuitry. Alternatively, some or all of the cache for the processor set may be located "off-chip". In some computing environments, the processor set 110 may be designed to work with qubits and perform quantum computing.

[0056] Computer-readable program instructions typically cause the processor set 110 of computer 101 to execute a series of operational steps, thereby loading them onto computer 101 to implement a computer implementation method, and thereby the instructions thus executed instantiate the methods specified in the flowcharts and / or descriptions of the computer implementation methods contained herein (collectively referred to as the "Methods of the Invention"). These computer-readable program instructions are stored in various types of computer-readable storage media, such as a cache 121 and other storage media considered below. The program instructions and associated data are accessed by the processor set 110 to control and direct the execution of the Methods of the Invention. In the computing environment 100, at least some of the instructions for executing the Methods of the Invention may be stored in a decontamination system 200 within persistent storage 113.

[0057] The communication fabric 111 is a signal conduction path that enables various components of the computer 101 to communicate with one another. Typically, this fabric is made up of switches and conductive paths, such as buses, bridges, physical input / output ports, and similar components. Other types of signal communication paths, such as fiber optic communication paths and / or wireless communication paths, may be used.

[0058] The volatile memory 112 is any type of volatile memory currently known or to be developed in the future. Examples include dynamic type random access memory (RAM) or static type RAM. Typically, volatile memory is characterized by random access, but this is not required unless explicitly stated. In computer 101, the volatile memory 112 is located in a single package and is internal to computer 101, but alternatively or additionally, the volatile memory may be distributed across multiple packages and / or located externally to computer 101.

[0059] The persistent storage 113 is any form of non-volatile storage for a computer that is currently known or may be developed in the future. The non-volatility of this storage means that the stored data is maintained regardless of whether power is supplied to the computer 101 and / or directly to the persistent storage 113. The persistent storage 113 may be read-only memory (ROM), but typically at least a portion of the persistent storage allows for writing, deleting, and rewriting of data. Some well-known forms of persistent storage include magnetic disks and solid-state storage devices. The operating system 122 may take several forms, such as various known proprietary operating systems or open-source portable operating system interface type operating systems that utilize a kernel. The code included in the decontamination system 200 typically includes at least some computer code involved in performing the methods of the present invention.

[0060] The peripheral device set 114 includes a set of peripheral devices for the computer 101. Data communication connections between the computer 101's peripheral devices and other components may be implemented in various ways, such as Bluetooth® connections, near-field communication (NFC) connections, connections formed by cables (such as Universal Serial Bus (USB) type cables), insert-type connections (e.g., Secure Digital (SD) cards), connections formed through local area communication networks, and even connections formed through wide area networks such as the Internet. In various embodiments, the UI device set 123 may include components such as a display screen, speakers, microphones, wearable devices (e.g., goggles and smartwatches), keyboards, mice, printers, touchpads, game controllers, and haptic devices. Storage 124 is external storage such as an external hard drive, or insertable storage such as an SD card. Storage 124 may be persistent and / or volatile. In some embodiments, storage 124 may take the form of a quantum computing storage device for storing data in the form of qubits. In embodiments where computer 101 requires a large amount of storage (for example, when computer 101 locally stores and manages a large database), this storage may be provided by peripheral storage devices designed to store very large amounts of data, such as a storage area network (SAN) shared by multiple geographically distributed computers. The IoT sensor set 125 consists of sensors that can be used in an Internet of Things application. For example, one sensor may be a thermometer and another may be a motion detector.

[0061] The network module 115 is a collection of computer software, hardware, and firmware that enables computer 101 to communicate with other computers via the WAN 102. The network module 115 may include hardware such as a modem or Wi-Fi® signal transceiver, software for packetizing and / or depackaging data for communication network transmission, and / or web browser software for communicating data over the Internet. In some embodiments, the network control and network forwarding functions of the network module 115 are performed on the same physical hardware device. In other embodiments (e.g., embodiments utilizing software-defined networking (SDN)), the control and forwarding functions of the network module 115 are performed on physically separate devices, resulting in the control function managing multiple different network hardware devices. Computer-readable program instructions for performing the method of the present invention can typically be downloaded to computer 101 from an external computer or external storage device through a network adapter card or network interface included in the network module 115.

[0062] WAN102 is any wide area network (e.g., the Internet) capable of transmitting computer data over non-local distances by any currently known or future-developed technology for transmitting computer data. In some embodiments, the WAN may be replaced and / or complemented by a local area network (LAN), such as a Wi-Fi network, designed to transmit data between devices located in a local area. WANs and / or LANs typically include computer hardware such as copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and edge servers.

[0063] The end-user device (EUD) 103 is any computer system used and controlled by an end-user (e.g., a customer of the company operating computer 101) and can take any of the forms considered above in relation to computer 101. The EUD 103 typically receives useful and valuable data from the operation of computer 101. For example, in a hypothetical case where computer 101 is designed to provide recommendations to an end-user, these recommendations are typically communicated from computer 101's network module 115 to the EUD 103 via the WAN 102. In this way, the EUD 103 can display or otherwise present the recommendations to the end-user. In some embodiments, the EUD 103 may be a client device such as a thin client, heavy client, mainframe computer, or desktop computer.

[0064] The remote server 104 is any computer system that serves at least some data and / or functions to computer 101. The remote server 104 may be controlled and used by the same entity that operates computer 101. The remote server 104 represents a machine that collects and stores useful and valuable data for use by other computers, such as computer 101. For example, in a hypothetical case where computer 101 is designed and programmed to provide recommendations based on historical data, this historical data may be provided to computer 101 from the remote database 130 of the remote server 104.

[0065] The public cloud 105 is any computer system available for use by multiple entities, providing on-demand availability of computer system resources and / or other computer functions, particularly data storage (cloud storage) and computing power, without direct active management by the user. Cloud computing typically leverages resource sharing to achieve coherence and economies of scale. Direct and active management of the computing resources of the public cloud 105 is performed by the computer hardware and / or software of the cloud orchestration module 141. The computing resources provided by the public cloud 105 are typically implemented by virtual computing environments running on various computers that make up the computers of the host physical machine set 142, which is the universe of physical computers in and / or available to the public cloud 105. The virtual computing environment (VCE) typically takes the form of virtual machines from the virtual machine set 143 and / or containers from the container set 144. These VCEs may be stored as images and are understood to be transportable either as images or after instantiation of the VCEs, among and between hosts of various physical machines. The cloud orchestration module 141 manages the transfer and storage of images, deploys new VCE instances, and manages active instances of VCE deployments. The gateway 140 is a collection of computer software, hardware, and firmware that enables the public cloud 105 to communicate through the WAN 102.

[0066] Here, some further explanation of virtualized computing environments (VCEs) is provided. A VCE can be stored as an "image." A new active instance of a VCE can be instantiated from an image. Two well-known types of VCEs are virtual machines and containers. A container is a VCE that uses operating system-level virtualization. This refers to an operating system feature where the kernel allows for the existence of multiple isolated user-space instances called containers. These isolated user-space instances typically behave like a real computer from the perspective of the program running within them. A computer program running on a normal operating system can utilize all of that computer's resources, including connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, a program running within a container can only use the contents of the container and the devices allocated to the container; this is a feature known as containerization.

[0067] The private cloud 106 is similar to the public cloud 105, except that its computing resources are available for use only by a single enterprise. Although the private cloud 106 is shown communicating with the WAN 102, in other embodiments the private cloud may be completely isolated from the internet and accessible only through a local or private network. A hybrid cloud is a combination of multiple clouds of different types (e.g., private, community, or public cloud types), often implemented by different vendors. Each of the multiple clouds remains a separate, discrete entity, but the larger hybrid cloud architecture is coupled together by standardized or proprietary technologies that enable orchestration, management, and / or data / application portability between the multiple configuration clouds. In this embodiment, both the public cloud 105 and the private cloud 106 are part of a larger hybrid cloud.

[0068] As shown in Figure 1, one or more computing devices, for example, a computer 101 or a remote server 104, may be specifically configured to implement a decontamination system 200 that operates to decontaminate a training dataset with respect to benchmark data. The configuration of the computing devices may include providing application-specific hardware, firmware, or the like to facilitate the performance of the operations and generation of outputs described herein with respect to exemplary embodiments. The configuration of the computing devices may also, or alternatively, include providing software applications stored in one or more storage devices and loaded into the memory of a computing device such as a computer 101 or a remote server 104, to cause one or more hardware processors of the computing devices to run software applications that configure the processors to perform the operations and generate outputs described herein with respect to exemplary embodiments. Furthermore, any combination of application-specific hardware, firmware, software applications running on hardware, or the like may be used without departing from the spirit and scope of the exemplary embodiments.

[0069] When a computing device is comprised of one of these methods, it should be understood that the computing device becomes a specialized computing device specifically configured to implement the mechanism of the exemplary embodiment, and not a general-purpose computing device. Furthermore, as will be discussed later, the implementation of the mechanism of the exemplary embodiment provides useful and concrete results that improve the functionality of the computing device and facilitate improved training of the code-generating LLM by decontaminating the training dataset with benchmark data. By removing the benchmark data from the training dataset, the trained code-generating LLM may be evaluated against the benchmark data to obtain an accurate representation of the code-generating LLM's performance.

[0070] Figure 2 is an example block diagram showing the main operating components of a decontamination system for a code-generating LLM training dataset, according to one exemplary embodiment. The operating components shown in Figure 2 may be implemented as dedicated computer hardware components, computer software running on computer hardware configured to perform specific computer operations resulting from those components, or any combination of dedicated computer hardware and computer software-configured computer hardware. It should be understood that these operating components perform automatically assigned operations without human intervention, even if the input may be provided by a human, such as a search query, and the resulting output may be useful to a human. The present invention particularly targets automatically operating computer components that improve the way a code-generating LLM is trained and provide a specific solution for implementing a decontamination mechanism for identifying content-sharing data between benchmark datasets and training datasets, identifying repositories potentially contaminated with benchmark data, and removing benchmark data from training datasets based on metadata that identifies content-sharing data and specifies the repository sources of the training data, which cannot be practically performed by a human as a mental process and does not target the organization of any human activity.

[0071] As shown in Figure 2, the decontamination system for the training dataset of the code generation LLM, also simply referred to as the decontamination system 200, includes a content sharing data identification engine 210, a contaminated repository identification engine 220, a contaminated training data identification engine 230, and a decontamination engine 240. The decontamination system 200 communicates with a benchmark data source computing system 250, a training data source computing system 260, and a code repository (or simply a repository) 270 via one or more data networks 202. Furthermore, the obtained decontaminated training dataset may be provided to the code generation LLM training engine 280 via one or more data networks 202 to train the code generation LLM 290.

[0072] According to one or more exemplary embodiments, the decontamination system 200 operates to decontaminate a training dataset, e.g., training dataset 262, with respect to having training data samples on which benchmark data exists, such as benchmark data samples from benchmark data 252. The decontamination system identifies repositories 270 containing repository data that matches content shared between benchmark data 252 and training data 262, and determines that these repositories are contaminated repositories. The metadata of the training data samples in training data 262 is then evaluated against the attributes of the contaminated repositories and benchmark datasets to determine which training data samples to remove from the training dataset, e.g., training data samples originating from contaminated repositories, by skipping string matching if there are differences in programming language between the benchmark data and training data samples and similar. String matching may also be performed to identify content-sharing data to remove from the training dataset. In some exemplary embodiments, metadata and string matching may be used to distinguish between true positive and false positive content-sharing data instances and to identify which to decontaminate. The results show that the mechanism of the exemplary embodiment can minimize or eliminate the identification of false negatives and false positives of contamination in the training dataset, and can also reduce the computational cost of decontamination of the training dataset.

[0073] In the mechanism of the exemplary embodiment shown in Figure 2, the decontamination system 200 may be invoked from the training data source computing system 260 to perform decontamination of a given training dataset 262. For example, the decontamination system 200 may be invoked to perform decontamination of the training dataset 262 before the code generation LLM training engine 280 uses the training dataset 262 to train the code generation LLM 290. The decontamination system 200 accesses the available benchmark dataset 252 from the benchmark data source computing system 250 and the training dataset 262 from the training data source computing system 260 via one or more data networks 202. The decontamination system 200 runs the content-sharing data identification engine 210 on the benchmark dataset 252 and the training dataset 262 to identify content-sharing data, i.e., instances of strings in the benchmark dataset 252 that are also present in the training dataset 262. The string matching engine 214 of the content-sharing data identification engine 210 performs string matching between datasets 252 and 262 to identify strings common to both.

[0074] If a string in benchmark dataset 252 exists in training dataset 262, these common strings, referred to as content-sharing data, are considered potential contamination, and the corresponding benchmark dataset ID of the benchmark dataset 252 that the string matches is assigned to the training data sample where the content-sharing data resides. Therefore, the training data sample metadata includes annotations that specify which benchmark datasets in benchmark dataset 252 have the matching string. These annotations also identify training data samples that are candidates for removal as part of the decontamination process, as described later.

[0075] Before the string matching engine 214 performs actual string matching between the benchmark dataset 252 and the training data 262, the normalization engine 212 may perform normalization operations on the benchmark dataset 252 and the training data 262. These normalizations may include whitespace removal, removal of special characters (e.g., newline characters), case-to-case conversion, and similar operations. In addition, the string matching engine 214 may implement other techniques to enhance the string matching performed, including the exclusion of short strings with lengths below a given threshold, thereby eliminating frequently used code snippets that may result in false positives. Furthermore, before performing string matching, the string matching engine 214 may perform comment removal to remove comment lines in the code of the samples from the benchmark dataset 252 and the training data 262.

[0076] To reduce the computational cost of string matching, the metadata evaluation engine 216 of the content sharing data identification engine 210 may evaluate the metadata of the benchmark dataset 252 and the training data 262 to determine when string matching can be skipped. That is, metadata attributes between datasets 252 and 262 may be evaluated to determine instances where string matching does not produce good results. For example, the metadata of datasets 252 and 262 may specify a particular programming language for data samples within datasets 252 and 262, and if there is a mismatch, for example, if the programming languages ​​of the benchmark data and the training data are different, string matching by the string matching engine 214 may be skipped for that portion of the training dataset 262. In addition, the data cleansing engine 218 may operate to clean the training dataset 262 by performing deduplication of training data samples in the training dataset 262. Furthermore, the data cleansing engine 218 may perform string deduplication on the benchmark dataset 252 so that, if there are duplicate strings within the same benchmark dataset 252, they are merged into a single string for the purpose of string matching with the training dataset 262.

[0077] The content-sharing data identification engine 210 thus identifies content-sharing data present in the pair of benchmark dataset 252 and training dataset 262. The content-sharing data may be maintained in data structure 219 for use in identifying contaminated repositories. That is, the content-sharing data is used to search repository data 272 of repository 270 for instances of content-sharing data (data having matching strings between benchmark dataset 252 and training dataset 262) to determine which repository 270 is potentially contaminated with benchmark data. As mentioned above, the repository stores documents used by code-generating LLMs, such as code-generating LLM 290, to perform code-generating operations. The documents may have various types of content, including actual computer code in various programming languages, code descriptions in natural language text, computer code issues and commits, and the like.

[0078] The content-sharing data matching engine 222 of the contaminated repository identification engine 220 performs a certain type of string matching between the content-sharing data identified by the content-sharing data identification engine 210, for example, stored in the data structure 219, and the repository data 272 of the repository. Instances of content-sharing data within the content of the repository data 272 may be maintained to determine the contamination metrics of the repository 270. These contamination metrics may be a numerical representation of the number of instances of content-sharing data in the repository data 272, the number of unique instances of content-sharing data in the repository data 272, the proportion of the repository data 272 that has content-sharing data, a function of the ratio of documents with content-sharing data to the total number of documents in the repository, or any other suitable metric for measuring the degree to which the repository data 272 is contaminated with content-sharing data. In this way, the repository 270 is evaluated for how much content is common between the repository data 272, the benchmark dataset 252, and the training dataset 262.

[0079] In some exemplary embodiments, the contamination metric may further include an evaluation of a function of the length of content-sharing data string instances present in the repository data 272. That is, longer strings represent a greater degree of contamination than shorter strings, where shorter strings are also more likely to be false positives. For this reason, different contamination metric values ​​may be associated with the repository data 272 depending on the length of the matched content-sharing data.

[0080] The repository scoring and ranking engine 224 operates to score and rank repositories 270 based on contamination metrics. Examples of such repository scoring and ranking are provided above, but any suitable scoring and ranking method may be used to identify potentially contaminated repositories. Based on the scoring and / or ranking of the frequency of occurrence of content-sharing data, a list of repositories, i.e., a repository list data structure 226, and scores / ranks may be generated, indicating which repositories are more / less contaminated with benchmark data. Thresholds may be established to determine when the degree of potential contamination is such that a repository 270 should be considered contaminated for the purpose of decontaminating the training dataset 262.

[0081] In some exemplary embodiments, the metadata evaluation engine 323 of the contaminated training data identification engine 230 utilizes the identification of the contaminated repository 270 to evaluate the metadata of training data samples in the training dataset 262 and identify those training data samples originating from the contaminated repository 270. In such embodiments, any instance of a training data sample originating from the contaminated repository 270 may be considered contaminated and may be flagged for decontamination. For this reason, comparing the metadata of the training data samples and the list of contaminated repositories 270 in this way may be performed before any other string matching is performed between the benchmark dataset 252 and the training dataset 262 for subsequent evaluation of the training dataset to identify potential contamination. That is, in some exemplary embodiments, the metadata evaluation engine 232 identifies any training data samples originating from the contaminated repository 270 (identified by the mechanism of the exemplary embodiment) and flags them for decontamination as part of a decontamination operation. Furthermore, string matching between the benchmark dataset 252 and the remaining portion of the training dataset 262, such as that performed by the string matching engine 214, may then be performed subject to the aforementioned attribute evaluation and other optimizations to speed up string matching, such as normalization, skipping string matching for different programming languages, and so on. This reduces the overhead of string matching operations by targeting only a subset of the training dataset for string matching, thereby reducing computational costs.

[0082] In these embodiments, metadata evaluation by the metadata evaluation engine 232 and string matching of the rest of the training data result in an annotated training dataset having annotations indicating the portion of the training dataset originating from the contaminated repository and the portion of the training dataset otherwise containing content-shared data. The annotated training dataset is then input to the decontamination engine 240, which may remove the flagged portions of the training dataset from the training dataset, thereby generating a decontaminated training dataset 242. The decontaminated training dataset 242 is a reduced-size training dataset 242 from which potentially contaminated training data samples have been removed.

[0083] The decontaminated training dataset 242 may then be stored in the training dataset source computing system 260 for future use in training a code-generating LLM such as the code-generating LLM 290. The decontaminated training dataset 242 may also be provided to the code-generating LLM training engine 280 for use when training the code-generating LLM 290. The decontaminated training dataset 242 does not contain training data samples that originate from the contaminated repository or have strings that match strings in the benchmark dataset 252. Therefore, once the code-generating LLM 290 is trained, its training is not biased towards the benchmark dataset 252. Thus, the benchmark dataset 252 may be used to accurately test the trained code-generating LLM 290 and accurately evaluate its performance.

[0084] The exemplary embodiments described above may be used to reduce instances of false negatives in content matching by assuming that all training data originating from a contaminated repository is actually contaminated in the training dataset. This may also apply to forked repositories originating from a contaminated repository, since forked repositories can also be assumed to be contaminated. In such embodiments, the training dataset may be decontaminated not only by removing instances of content-sharing data detected based on string matching between the benchmark data and the training dataset, but also based on metadata specifying the repository source of the training data. That is, for training data samples originating from a contaminated repository, those training data samples may be removed from the training dataset without the need to perform string matching between the benchmark data and the training dataset. Then, the remaining training dataset samples may be subjected to string matching between the benchmark dataset and the training dataset to identify instances of content-sharing data, which may then be removed from the training dataset.

[0085] Therefore, once the contaminated repository list data structure 226 is generated, the first step in the subsequent decontamination operation of the training dataset may be to compare the metadata of the training data samples with the list to identify which training data samples originate from the contaminated repository and remove them from the training dataset. The remaining training dataset is then subjected to an optimized string matching operation to identify the remaining potentially contaminated portions, and these additional portions may also be removed as part of the decontamination.

[0086] In other exemplary embodiments, a repository list data structure 226 containing content-sharing data to a degree sufficient to be considered potentially contaminated, i.e., contaminated for the purpose of decontamination of the training dataset, may be used to identify which portions of the training dataset 262, flagged as potentially contaminated by string matching operations for identifying content-sharing data, are actual contaminated training data by benchmark data, and which may be false positives. That is, in these exemplary embodiments, the mechanism of the content-sharing data identification engine 210 identifies matching string instances between the benchmark dataset 252 and the training dataset 262. These are potentially contaminated and may contain false positives. Thus, in order to distinguish between false positives and true positives, these training data samples flagged as potentially contaminated may be further evaluated against the repository list data structure 226, and if they originate from a contaminated repository, they are considered true positives and may be removed through decontamination. Any instances not originating from a contaminated repository may be considered false positives, not subjected to decontamination, and may remain in the training dataset. Therefore, a more granular evaluation of potentially contaminated training datasets may be performed to distinguish between false positives and true positives.

[0087] As can be seen from the above, exemplary embodiments provide computer tools and computer operations / functions for decontaminating a training dataset with respect to benchmark data. This may be done using a combination of metadata-based identification and string matching of training data samples originating from identified contaminated repositories. In some exemplary embodiments, false positives and true positives may be distinguished based on such metadata and identification of contaminated repositories. The resulting decontaminated training dataset may then be provided to the training logic for training a code-generating LLM. As a result, the trained code-generating LLM is not trained on benchmark data, and therefore the benchmark dataset can be accurately used to evaluate the performance of the trained code-generating LLM.

[0088] As described above, the exemplary embodiments operate on benchmark data samples and repository data, which may include metadata describing various attributes of the repository data. Figure 3A is a diagram of one example of a benchmark data sample according to one exemplary embodiment. Figure 3A shows two examples of benchmark data from a benchmark dataset. In the first example 310, the benchmark data sample includes a prompt 312 and a normative answer 314. The prompt 312 specifies a problem that the code generation LLM should solve by generating code to perform the requested action in the prompt 312. The normative answer includes the code that the code generation LLM should generate to solve the problem specified in the prompt 312. The strings of the prompt 312 and the normative answer 314 may be used for string matching against a training dataset to identify content-sharing data that may have a similar format.

[0089] In the second example, 320, the benchmark data sample includes input text 322 in which the user specifies in natural language the task they want the code generation LLM to perform, for example, "Write a Python function that identifies nonprime numbers." Code 324 is the result that should be generated by the code generation LLM to satisfy the request of text 322. In this example, too, the strings may be used to identify content-sharing data by performing string matching against the training dataset. The exemplary prompt 312 in Figure 3A includes a portion of Python code for importing modules and defining a function prototype, as well as explanatory instructions expressed as comments on the Python code. Instead of using these entire strings for string matching, the instruction portions embedded within the comments (within the briefed boxes) are used for string matching against the training data. In Example 320, the text and code are used as they are for these purposes.

[0090] Figure 3B is a diagram of an example of a repository record with corresponding metadata according to one exemplary embodiment. As shown in Figure 3B, repository record 330 includes several metadata fields 332, 334, 336, and content 338. Metadata field 332 specifies the repository name of the repository in which the record resides. Metadata field 334 specifies the file path of content 338, and metadata field 336 specifies the programming language of content 338. This same metadata is carried with any training data samples generated from this record and associated with those training data samples in the training dataset. For this reason, metadata field 336 may be used to determine when to skip string matching based on differences in programming language. Furthermore, metadata field 332 may be used to identify portions of the training dataset originating from this repository, for example, if the repository is determined to be contaminated.

[0091] Figure 4 is a flowchart outlining the operation of an example of a decontamination system according to one exemplary embodiment. It should be understood that the operations outlined in Figure 4 are performed automatically and specifically by the improved computer tools of the exemplary embodiment and are not intended to be performed by humans as mental processes or by organizing human activities, and are not practically feasible to do so. On the contrary, while humans may, in some cases, initiate the execution of the operations described in Figure 4, and in some cases utilize the results generated as a consequence of the operations described in Figure 4, the operations in Figure 4 themselves are performed specifically by improved computing tools in an automated manner.

[0092] As shown in Figure 4, the operation begins by receiving one or more benchmark datasets and the training dataset to be decontaminated (step 410). A content matching operation is performed between the benchmark dataset and the training dataset to identify content-shared data (step 420). Instances of content-shared data in the training dataset are annotated with an identifier of the benchmark dataset having one or more matching strings (step 430). The content-shared data is stored and compared with repository data to identify repositories that have instances of the content-shared data (step 440). Based on the instances of content-shared data in the repositories, contamination metrics are generated for each repository (step 450). Repositories are scored and ranked based on their contamination metrics (step 460). Based on the scoring and ranking, a list of contaminated repositories is generated (step 470). The list of contaminated repositories is used to match against the metadata of training data samples in the training dataset to identify portions originating from contaminated repositories (step 480). Next, the training dataset is decontaminated based on identified portions originating from the contaminated repository, and further based on string matching between benchmark datasets (step 490). The decontaminated training dataset is then fed into the code generation LLM training engine for training the code generation LLM (step 500). The operation then terminates.

[0093] The description of the present invention is presented for illustrative and explanatory purposes and is not intended to be exhaustive or to limit the invention to the disclosed forms. Many modifications and variations will become apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The embodiments have been selected and described to best illustrate the principles and practical applications of the invention and to enable those skilled in the art to understand the invention in various embodiments with various modifications to suit specific uses intended. The terminology used herein has been selected to best illustrate the principles, practical applications, or technological improvements over existing technologies, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. The step involves performing content matching between benchmark data and training data to identify content-shared data, where the content-shared data includes strings that are common to both the benchmark data and the training data; A step of identifying one or more repositories containing instances of the aforementioned content sharing data; A step of generating one or more pollution metrics for each repository in the one or more repositories based on the frequency of instances of the content sharing data in the one or more repositories; A step of identifying one or more contaminated repositories within the one or more repositories based on the one or more contaminated metrics for each repository; A step of decontaminating the training data based on the identification of the one or more contaminated repositories and the metadata of the training data samples in the training data; and The next step is to train a machine learning computer model based on the aforementioned decontaminated training data. A method for providing this.

2. The step of decontaminating the training data is performed for each training data sample within the training data. A step of comparing one or more first attributes of the training data sample in the metadata of the training data sample with one or more second attributes of the one or more contaminated repositories; and Steps to remove the training data sample from the training data in response to identifying a match between at least one first attribute of the training data sample and at least one second attribute of one or more contaminated repositories. The method according to claim 1, comprising:

3. The method according to claim 2, wherein the one or more first attributes include an identifier for a source repository from which the corresponding training data samples were obtained, and the one or more second attributes include a repository identifier for a corresponding contaminated repository.

4. The method according to claim 1, wherein the step of decontaminating the training data includes a step of performing preprocessing on the training data to identify portions of the training data that are not subject to decontamination, and the preprocessing includes identifying portions of the training data in a computer programming language different from the programming language of the benchmark data.

5. The method according to claim 1, wherein the one or more contamination metrics include at least one of the number of instances of content-sharing data in the one or more repositories, or the number of unique instances of content-sharing data in the one or more repositories.

6. The method according to claim 1, wherein the one or more contamination metrics include the degree of contamination of the corresponding repository, the degree of contamination is a function of the ratio of the number of documents in the corresponding repository having instances of the content sharing data to the total number of documents in the corresponding repository.

7. The method according to claim 1, wherein the step of performing content matching between benchmark data and training data to identify the content sharing data further comprises the step of annotating training data samples in the training data that have the content sharing data with benchmark annotations specifying identifiers of the benchmark data, and the training data samples annotated with benchmark annotations are candidates for removal from the training data.

8. The method according to claim 1, wherein the one or more pollution metrics include a function of the length of content-sharing data string instances present in the corresponding repository.

9. The method according to claim 1, wherein the machine learning computer model is a code generation large-scale language model (LLM).

10. The method according to claim 9, wherein the one or more repositories are code generation repositories containing documents used by the code generation LLM to generate computer code for performing a function specified in a prompt input to the code generation LLM.

11. In the processor, A procedure for performing content matching between benchmark data and training data, thereby identifying content-shared data, wherein the content-shared data includes strings that are common to both the benchmark data and the training data; A procedure for identifying one or more repositories containing instances of the aforementioned content sharing data; A procedure for generating one or more pollution metrics for each repository in the one or more repositories, based on the frequency of instances of the content-sharing data in the one or more repositories; A procedure for identifying one or more contaminated repositories within the one or more repositories based on the one or more contaminated metrics for each repository; A procedure for decontaminating the training data based on the identification of the one or more contaminated repositories and the metadata of the training data samples in the training data; and Procedure for training a machine learning computer model based on the aforementioned decontaminated training data. A computer program designed to execute something.

12. The procedure for decontaminating the training data is performed for each training data sample within the training data. A procedure for comparing one or more first attributes of the training data sample in the metadata of the training data sample with one or more second attributes of the one or more contaminated repositories; and A procedure to remove the training data sample from the training data in response to identifying a match between at least one first attribute of the training data sample and at least one second attribute of one or more contaminated repositories. The computer program according to claim 11, including the computer program described in claim 11.

13. The computer program according to claim 12, wherein the one or more first attributes include an identifier for a source repository from which the corresponding training data sample was obtained, and the one or more second attributes include a repository identifier for a corresponding contaminated repository.

14. The computer program according to claim 11, wherein the procedure for decontaminating the training data includes a step of performing preprocessing on the training data to identify portions of the training data that are not subject to decontamination, and the preprocessing includes identifying portions of the training data in a computer programming language different from the programming language of the benchmark data.

15. The computer program according to claim 11, wherein the one or more contamination metrics include at least one of the number of instances of content-sharing data in the one or more repositories, or the number of unique instances of content-sharing data in the one or more repositories.

16. The computer program according to claim 11, wherein the one or more contamination metrics include the degree of contamination of a corresponding repository, the degree of contamination being a function of the ratio of the number of documents in the corresponding repository having instances of the content sharing data to the total number of documents in the corresponding repository.

17. The computer program according to claim 11, wherein the procedure for performing content matching between benchmark data and training data to identify the content-sharing data further comprises the procedure for annotating training data samples in the training data that have the content-sharing data with benchmark annotations specifying identifiers of the benchmark data, and the training data samples annotated with benchmark annotations are candidates for removal from the training data.

18. The computer program according to claim 11, wherein the one or more pollution metrics include a function of the length of content-sharing data string instances present in the corresponding repository.

19. The computer program according to claim 11, wherein the machine learning computer model is a code-generating large-scale language model (LLM), and the one or more repositories are code-generating repositories containing documents used by the code-generating LLM to generate computer code for performing a function specified in a prompt input to the code-generating LLM.

20. Processor set; One or more computer-readable storage media; and Program instructions stored on the one or more computer-readable storage media, wherein the processor set A procedure for performing content matching between benchmark data and training data, thereby identifying content-shared data, wherein the content-shared data includes strings that are common to both the benchmark data and the training data; A procedure for identifying one or more repositories containing instances of the aforementioned content sharing data; A procedure for generating one or more pollution metrics for each repository in the one or more repositories, based on the frequency of instances of the content-sharing data in the one or more repositories; A procedure for identifying one or more contaminated repositories within the one or more repositories based on the one or more contaminated metrics for each repository; A procedure for decontaminating the training data based on the identification of the one or more contaminated repositories and the metadata of the training data samples in the training data; and Procedure for training a machine learning computer model based on the aforementioned decontaminated training data. Program instructions for causing an operation to be performed A computer system equipped with the following features.

Images