Test check of pull request changes using large language models

Abstract

A large language model trained on source code and natural language text predicts the test necessity of changes in pull requests of a code library. The cues provided to the large language model include questions and answer formats for the model to predict whether a unit test needs to be performed for a change. The problem describes tasks of the model, changes made to the file, and content of the changed file. When the model predicts that a change requires a unit test, a check is made to determine whether a test exists. An annotation is inserted into the pull request indicating a changed test necessity and whether a unit test for the test exists.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of an earlier filed provisional application, serial number 63 / 452,667, filed on March 16, 2023, entitled “AI Driven Test Checking,” the entire contents of which are incorporated herein by reference. Background Art

[0003] A version control repository stores each version of an artifact, such as source code files, test files, documentation files, and so on, and tracks changes between different versions. Repositories managed by a version control system are typically distributed, so that every user of the repository has a working copy of the repository. A version control system coordinates the distribution of changes to the repository's contents to different users.

[0004] When a user finishes editing a source code file, the user performs a commit, which checks the modified version of the file back into the shared file repository. A pull request notifies others that changes have been made to the file that are pushed or committed back to the repository.

[0005] It's common practice to test changes before submitting them and include the tests in pull requests. Some repositories enforce unit testing by calculating a test coverage metric based on the percentage of lines run by unit tests. If the proposed code changes fail to meet the test coverage threshold, the pull request is rejected.

[0006] However, sometimes, test coverage based on the percentage of lines exercised in unit tests is inversely proportional, as not all changes require tests. Small changes, changes that are difficult to test, or changes incidental to code functionality generally do not require additional tests. Test coverage requirements can sometimes delay and unnecessarily burden code review cycles. Summary of the Invention

[0007] This Summary is provided to introduce some concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0008] An automated system detects test necessity for changes in pull requests and, when a change is test-necessary, checks for unit tests to cover the change. A change can be a newly created file or a change that modifies the contents of a file. Test necessity is predicted using a large language model trained on natural language text and source code. This model provides prompts for the prediction, including the expected response format and question format. When a change is predicted to be test-necessary, existing unit tests are checked to test the change. The prediction and presence of unit tests are recorded in the pull request.

[0009] These and other features and advantages will be apparent from a reading of the following detailed description and a review of the associated drawings.It is to be understood that both the foregoing general description and the following detailed description are explanatory only and are not restrictive of the aspects, as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 An exemplary system for using a large language model to determine the necessity of testing code changes is shown.

[0011] Figure 2 Exemplary prompting for a large language model to determine the necessity of testing a new document is shown.

[0012] Figure 3 is a flow chart illustrating an exemplary method of determining code changes.

[0013] Figure 4 is a block diagram illustrating an exemplary operating environment. DETAILED DESCRIPTION

[0014] Overview

[0015] The disclosed subject matter relates to using large language models to predict the test necessity of changes made to files in a version-controlled source code repository. Test necessity changes are those changes that need to be tested. Not all changes in a pull request need to be tested. Small changes, changes that are difficult to test, changes that are not related to the code functionality but are not major (such as changes to code comments or code formatting), or changes that affect the command-line interface do not need to have tests added. Software vulnerability fixes should be tested to ensure that the vulnerability does not recur. The technology described in this article uses large language models to predict whether changes identified in pull requests need to be tested. Large language models are trained on large training datasets of natural language text and source code to learn patterns in the data, which are used to predict whether changes need to be tested.

[0016] On the one hand, the technique applies to newly created files and changes that alter the content of existing files. Changes that delete files are never considered worthy of testing. The technique generates hints for the large language model, which include the necessary context for the large language model to predict the necessity of testing newly created files or changes made to existing files.

[0017] Program changes can be tested using a variety of tests, including, but not limited to, unit tests, integration tests, regression tests, acceptance tests, smoke tests, and security tests. Unit tests test the smallest logically isolated unit of code in a program. Integration tests test a set of software modules as a group. Smoke tests test the overall functionality of a program. Regression tests rerun functional and non-functional tests after a change to ensure that the program operates as expected. Acceptance tests ensure that the end-user's business requirements are met. Changes in pull requests are tested using unit tests.

[0018] Attention will now be directed to systems, apparatus, and methods for testing inspection systems.

[0019] system

[0020] Figure 1 An exemplary system 100 for predicting the necessity of testing changes from pull requests is shown. System 100 includes a code repository 102, a pull request engine 106, a hint generation engine 112, a large language model 116, and a test checker 119. Pull request engine 106 interacts with code repository 102 and detects pull requests 104 with commits that identify changes to existing files or creation of new files.

[0021] Code repository 102 can be a file archive and web hosting facility for storing a large number of software development assets, such as, but not limited to, source code files, unit tests, script files, documentation, etc., whether private or public. Code repository 102 can be built as a version control system, such as GIT, Mercurial, etc. The files in code repository 102 are different and may be written in different programming languages.

[0022] Pull request engine 106 scans each pull request 104 uploaded to code repository 102. If pull request 104 indicates the creation of a new file, pull request engine 106 extracts the new file and file name 110, and hint generation engine 112 creates hints 114 based on these for large language model 116. If pull request 104 indicates changes to existing files, pull request engine 106 extracts the diff of each change and the changed file from code repository 102. Hint generation engine 112 creates hints 114 based on the changed diff and changed file 108 for large language model 116.

[0023] Diff is a utility program used to compare the differences between two files. The diff program displays the differences between two versions of a file, such as a file before and after a change. For example, Git diff is a command that runs functions on Git data sources (such as commits, branches, and files) to display the changes. The output of the diff program is a diff file that displays the changes in diff format. The diff format includes a "+" character to indicate additions to files and a "-" character to indicate deletions.

[0024] In one aspect, the large language model 116 is a neural transformer model with attention. The neural transformer model with attention is a unique type of machine learning model. Machine learning involves the use and development of computer systems that can learn and adapt without following explicit instructions, by using algorithms and statistical models to analyze data and infer conclusions from the data. Machine learning uses different types of statistical methods to learn from data and predict future decisions. Traditional machine learning includes classification models, data mining, Bayesian networks, Markov models, clustering, and visual data mapping.

[0025] Deep learning differs from traditional machine learning because it processes data in multiple stages through many hidden layers of a neural network to learn and interpret features and the relationships between features. Deep learning includes neural networks, unlike traditional machine learning techniques that do not use neural networks. The Neural Transformer model is a type of deep learning that uses attention. Attention guides the neural network to focus on certain features or tokens in the input sequence, thereby learning different representations from different locations of the token in the input sequence. The Neural Transformer model handles the dependencies between its input and output through attention, without using recurrent neural networks (RNNs) such as long short-term memory (LSTM) networks and convolutional neural networks (CNNs).

[0026] There are multiple configurations of Neural Transformer models with attention. In one aspect, a large language model is configured as an encoder-decoder Neural Transformer model with attention. An encoder-decoder Neural Transformer model with attention consists of a series of stacked encoder blocks coupled to a series of stacked decoder blocks. The encoder blocks extract features from the input sequence, generating embeddings for each token in the input sequence, and the decoder uses these features to generate an output sentence representing a candidate for a repair code. In another aspect, a large language model can be a decoder Neural Transformer model with attention, with only a decoder block.

[0027] Large language models are pre-trained on natural language text and source code. Examples of neural transformer models with attention include OpenAI's Codex and other neural transformer models with attention provided by OpenAI, Google, Facebook, and Microsoft. On the one hand, large language models are hosted on external servers and accessed over the network through application programming interfaces (APIs). Training large language models requires a large amount of training data and computing resources, which prevents some developers from creating their own models. Instead, third parties that create large language models may provide developers with access to large language models in the form of cloud services, which may be charged or free.

[0028] The context window is the number of tokens a large language model can process in a forward pass. These tokens represent the natural language text and source code included in the prompt. The size of the context window varies, for example, OpenAI's Codex has a context window size of 4,096 tokens, and OpenAI's GPT-4 model has a context window size of 32,000 tokens. The prompt is limited by the context window size of the large language model.

[0029] The prompt generation engine 112 creates prompts 114 for changes made to source code files. In one aspect, the prompts can be formatted in a conversational manner, including questions 120 and answer formats 122. Questions 120 include a task description 124, a description of the changes made to the file 126, and a query 129. The task description 124 provides the model with background information about the nature of the task in natural language. Following the task description 124 is a diff file 130 showing the changes made to the file, a file instruction 132 after the changes were made, and the file content 134 after the changes were made. The file instruction after the changes indicates that the prompt includes the file content after the changes were made.

[0030] The query 128 is a natural language instruction about the task to be performed and is immediately followed by the answer format 122. The answer format instructs the model to provide a response in the expected format, with the last line of the prompt being the phrase "Answer:".

[0031] On the one hand, the model can return a single number that is the log probability of being labeled "yes." A threshold is used to determine whether this number represents "yes" or "no."

[0032] It should be noted that the wording of the prompt is not limited to the exact wording shown in prompt 114. Other question and answer formats can be used. For example, the answer format can consist of a simple yes or no answer without any explanation. In addition, the answer format can require the model to return a completion and check whether it begins with the word "yes" instead of returning a log probability.

[0033] Test checker 119 receives answers 118 from large language model 116. When the answers indicate that a change requires testing, test checker 119 checks the pull request to see if there are existing tests for the change. If no such tests exist, the test checker posts a comment in the pull request suggesting that a test be created for the change. Alternatively, the test checker can issue an alert in the code repository, prompting the developer to add a test for the change. Alternatively, the test checker can call another application or system to create test suggestions and post these suggestions in the pull request.

[0034] Figure 2 This is an illustration of an exemplary prompt for a scenario where the change involves creating a new file. The prompt 200 includes a task description 202, a newly created file 204, a question 206, and an answer format 208. The task description 202 provides the model with background information about the nature of the task in natural language. The prompt includes the contents of the newly created file, or a portion of the file if the file is too long. The question 206 indicates the target task for the model, and the answer format 208 indicates the expected output of the model's question.

[0035] In some cases, the newly created file may be quite large and its entire content may not fit within the context window size of a large language model. In this case, the parts of the file that are considered less important are replaced with the replacement string, while the parts of the file that are considered important are included in the hint.

[0036] method

[0037] Attention will now be directed to the description of various exemplary methods utilizing the systems and devices disclosed herein. The operation of various aspects can be further described with reference to the various exemplary methods. It should be understood that the representative methods do not necessarily have to be performed in the order presented, or in any particular order, unless otherwise indicated. In addition, the various activities described with respect to these methods can be performed in a serial or parallel manner, or in any combination of serial and parallel operations. In one or more aspects, the method illustrates the operation of the systems and devices disclosed herein.

[0038] Figure 3 is a flow chart of a method 300 for determining the necessity of testing changes in a pull request using a large language model. Figure 1-Figure 3 The pull request engine 106 detects a pull request submitted to the code repository 102 (block 302). The pull request 104 includes one or more changes to files of the code repository. Each change identified in the pull request is processed (block 304).

[0039] If the change involves a newly created file (block 306), the new file is obtained. The new file is converted into tokens. A prompt is then created in question-answer format, including the contents of the new file, the new file name, and instructions for the model. The number of tokens in the prompt must fit within the context window size of a large language model. If the token size of the prompt exceeds the context window size, the prompt is analyzed to replace portions that are not considered important with replacement strings until the size of the new file fits within the context window.

[0040] If the change involves a change to an existing file (block 308), the prompt generation engine obtains the changed diff file and the changed file. A prompt is generated, including the task description, the changed diff file, the changed file content, the question, and the answer format (block 308). The number of tokens in the prompt must fit within the context window size of the large language model. If the token size of the prompt exceeds the context window size, the prompt is analyzed to replace the parts that are not considered important with the replacement string until the size of the new file fits within the context window.

[0041] The prompt is then applied to the large language model (block 310). In one aspect, the large language model is hosted on an external server and can be accessed through various application programming interfaces (APIs). The test checker 119 transmits the prompt via the API and receives a response (block 312). The response indicates a yes, indicating that the test is needed, or a no, indicating that the test is not needed. In some cases, the model can respond with an explanation.

[0042] If the model predicts that a test is necessary for the change, the test checker 119 receives the response and searches for existing tests for the change (block 314). If a test exists, the test checker 119 can insert a note in the pull request indicating that the test is needed and, if a test exists, annotate that an existing test is available. Alternatively, the test checker can notify the developer using the pull request or intuitively through the user interface of the code repository in any other possible way. Alternatively, the test checker collaborates with another application or system to create test suggestions and forwards these suggestions to the pull request.

[0043] Exemplary Operating Environment

[0044] Attention now turns to a discussion of an exemplary operating environment. Figure 4 An exemplary operating environment 400 is shown, wherein one or more computing devices 402 are used to generate hints for a large language model, and one or more computing devices 404 are used to host the large language model that uses the hints to predict responses. However, it should be noted that the aspects disclosed herein are not limited to any particular device configuration. Alternatively, a single computing device can host the large language model and generate hints based on changes to a pull request.

[0045] Computing devices 402 and 404 can be any type of electronic device, such as, but not limited to, a mobile device, a personal digital assistant, a mobile computing device, a smartphone, a cellular phone, a handheld computer, a server, a server array or server farm, a network server, a blade server, an Internet server, a workstation, a minicomputer, a mainframe computer, a supercomputer, a network device, a network appliance, a distributed computing system, a multiprocessor system, or a combination thereof. Operating environment 400 can be deployed in a network environment, a distributed environment, a multiprocessor environment, or a standalone computing device with access to remote or local storage devices.

[0046] Computing devices 402 and 404 may include one or more processors 408 and 430, one or more communication interfaces 410 and 432, one or more storage devices 412 and 432, one or more input / output devices 414 and 434, and one or more memory devices 416 and 438. Processors 408 and 430 may be any commercially available or custom processors and may include dual microprocessor and multi-processor architectures. Communication interfaces 410 and 432 facilitate wired or wireless communication between computing devices 402 and 404 and other devices. Storage devices 412 and 436 may be computer-readable media that do not include propagated signals (e.g., modulated data signals transmitted over a carrier wave). Examples of storage devices 412 and 436 include, but are not limited to, RAM, ROM, EEPROM, flash memory or other storage technologies, CD-ROM, digital versatile disks (DVDs) or other optical storage, magnetic cassettes, magnetic tape, and magnetic disk storage, all of which do not include propagated signals (e.g., modulated data signals transmitted over a carrier wave). Computing devices 402 and 404 may have multiple storage devices 412. Input / output devices 414 and 434 may include a keyboard, a mouse, a pen, a voice input device, a touch input device, a display, a speaker, a printer, etc., and any combination thereof.

[0047] Memory device or memory 416, memory device or memory 438 can be any non-transitory computer-readable storage medium that can store executable programs, applications, and data. The computer-readable storage medium is independent of the propagation signal (e.g., a modulated data signal transmitted via a carrier wave). It can be any type of non-transitory memory device (e.g., random access memory, read-only memory, etc.), magnetic storage, volatile storage, non-volatile storage, optical storage, DVD, CD, floppy disk drive, etc., independent of the propagation signal (e.g., a modulated data signal transmitted via a carrier wave). Memory device 416, memory device 438 can also include one or more external storage devices or remote storage devices that are independent of the propagation signal (e.g., a modulated data signal transmitted via a carrier wave).

[0048] Memory devices 416 and 438 may include instructions, components, and data. A component is a software program that performs a specific function, also known as a module, program, component, and / or application. Memory device 416 may include an operating system 418, a code library 420, a pull request engine 422, a hint generation engine 424, a test checker 426, and other applications and data 428. Memory device 438 may include an operating system 440, a large language model 442, and other applications and data 444.

[0049] The computing device 402 may be communicatively coupled via a network 406. The network 406 may be configured as an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a portion of a public switched telephone network (PSTN), a plain old telephone service (POTS) network, a wireless network, network, or any other type of network or combination of networks.

[0050] The network 406 may employ various wired and / or wireless communication protocols and / or technologies. The various generations of different communication protocols and / or technologies that the network may employ may include, but are not limited to, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access 2000 (CDMA-2000), High Speed ​​Downlink Packet Access (HSDPA), Long Term Evolution (LTE), Universal Mobile Telecommunications System (UMTS), Evolution-Data Optimized (EV-DO), Worldwide Interoperability for Microwave Access (WiMax), Time Division Multiple Access (TDMA), Orthogonal Frequency Division Multiplexing (OFDM), Ultra-Wideband (UWB), Wireless Application Protocol (WAP), User Datagram Protocol (UDP), Transmission Control Protocol / Internet Protocol (TCP / IP), any portion of the Open Systems Interconnection (OSI) model protocols, Session Initiation Protocol / Real-time Transport Protocol (SIP / RTP), Short Message Service (SMS), Multimedia Message Service (MMS), or any other communication protocols and / or technologies.

[0051] Technical Effects

[0052] Aspects of the subject matter disclosed herein relate to the technical problem of determining whether a change in a pull request requires testing before merging the change into the codebase. A technical feature associated with solving this problem is the generation of data that informs a large language model about whether the change requires testing. The resulting technical effect is an accurate determination of the need for testing a change without unduly increasing computational burden.

[0053] Those skilled in the art will understand that technical effects are the purpose of technical embodiments. The mere fact that an embodiment involves computation does not preclude the existence of a technical effect, nor does it change the specific and technical nature of the embodiment. The operations used to create prompts and interact with large language models herein are understood to be digital in nature. The human mind cannot directly interface with a CPU, network interface card, or other processor, or with RAM or other digital storage device, as disclosed herein, to read or write the necessary data and perform the necessary operations on digital values.

[0054] These embodiments are also assumed to be capable of being run under strict time constraints in a production environment, or at scale in a test lab within a production environment, and are not just thought experiments.

[0055] in conclusion

[0056] A system is disclosed, comprising: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for performing the following operations: accessing a large language model trained on source code and natural language to perform a task given a prompt; detecting changes to file content from pull requests in a code repository; constructing a prompt for the large language model to predict whether the file change requires testing, wherein the prompt includes a task description, a description of the change to the file, the content of the file after the change, and an answer format; and obtaining a response from the large language model given the prompt, the response predicting whether the file change requires testing.

[0057] In one aspect, if the response indicates a test change, determining whether a test exists from the pull request that executes the test. In one aspect, indicating in the pull request that the test is needed and whether the test is available in the repository. In one aspect, the description of the change to the file includes a diff file or diff format of the change.

[0058] In one aspect, the answer format includes a yes response format and a no response format. In one aspect, the answer format includes an explanation for a large language model to clarify the response. In one aspect, the large language model is a neural transformer model with attention.

[0059] A computer-implemented method is disclosed, comprising: detecting changes in a pull request for a file of a code repository; accessing a large language model trained on source code and natural language; creating a prompt for the large language model to predict whether the new file requires testing, wherein the prompt includes a task description, the content of the new file, and an answer format; and obtaining a response from the large language model, given the prompt, the response predicting whether the new file requires testing.

[0060] In one aspect, the computer-implemented method further comprises: when the response indicates that the new file requires testing, checking whether a test exists for the new file. In one aspect, the computer-implemented method further comprises: annotating in the pull request that the new file requires testing. In one aspect, the computer-implemented method further comprises: indicating in the pull request that a test exists for the new file.

[0061] In one aspect, the computer-implemented method further comprises: reducing a portion of the content of the new document to fit within a context window size of a large language model. In one aspect, the answer format comprises a yes or no response. In one aspect, the answer format comprises an interpretation of the response by the large language model. In one aspect, the large language model is a neural transformer model with attention.

[0062] A computer-implemented method is disclosed, comprising: obtaining a change from a pull request for a code repository, wherein the change alters file content or creates a new file; predicting testing necessity of the change using a large language model, wherein a prompt to the large language model includes a task description, the change, and an answer format; receiving a response from the large language model; and when the response indicates that the change is testing necessary: ​​checking whether the pull request includes tests that cover the change; and responding to the pull request with the response and whether the test is included in the pull request.

[0063] In one aspect, the computer-implemented method further comprises: when the response indicates that the change is not necessary for testing, providing a comment in the pull request indicating that the change is not necessary for testing. In one aspect, the computer-implemented method further comprises: generating suggested tests for the change. In one aspect, the computer-implemented method further comprises: reducing the size of the prompt to fit within the context window size of the large language model. In one aspect, the large language model comprises a neural transformer model with attention.

[0064] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

[0065] It should be understood that the representative methods do not necessarily have to be performed in the order presented, or in any particular order, unless otherwise indicated. In addition, the various activities described with respect to the methods can be performed in a serial or parallel manner, or in any combination of serial and parallel operations. In one or more aspects, the methods illustrate the operation of the systems and devices disclosed herein.

Claims

1. A system comprising: one or more processors; as well as A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following actions: Access large language models trained on source code and natural language to perform tasks given prompts; Detect changes to file contents from pull requests coming from the code repository; Constructing a prompt for the large language model to predict whether the change to the file requires testing, wherein the prompt includes: a task description, a description of the change to the file, the content of the file after the change, and an answer format; as well as Given the prompt, a response is obtained from the large language model predicting whether the change to the file requires testing.

2. The system of claim 1 , wherein the one or more programs include instructions to: If the response indicates to test the change, then determining whether a test exists from the pull request to perform the test.

3. The system of claim 2, wherein the one or more programs include instructions to: Indicate in the pull request that tests are required and whether the tests are available in the repository. 4 . The system of claim 1 , wherein the description of the changes to the file comprises a diff file or diff format of the changes. The system according to claim 1 , wherein the answer format includes a yes response format and a no response format.

6. The system of claim 1, wherein the answer format includes an explanation for the large language model to clarify the response.

7. The system of claim 1, wherein the large language model is a neural transformer model with attention.

8. A computer-implemented method comprising: Detect changes to files in the codebase in pull requests; Access to large language models trained on source code and natural language; Creating a prompt for the large language model to predict whether a new document requires testing, wherein the prompt includes: a task description, the content of the new document, and an answer format; as well as Given the prompt, a response is obtained from the large language model predicting whether the new document needs to be tested.

9. The computer-implemented method of claim 8, further comprising: When the response indicates that the new file requires testing, a check is made to see if a test exists to test the new file.

10. The computer-implemented method of claim 9, further comprising: Comment in the pull request that the new file needs testing.

11. The computer-implemented method of claim 10 , further comprising: The existence of the test for the new file is indicated in the pull request.

12. The computer-implemented method of claim 8, further comprising: A portion of the content of the new file is reduced to a size suitable for a context window of the large language model.

13. The computer-implemented method of claim 8, wherein the answer format comprises a yes or no response.

14. The computer-implemented method of claim 13, wherein the answer format comprises an interpretation of the response from the large language model.

15. The computer-implemented method of claim 8, wherein the large language model is a neural transformer model with attention.

Images