Fault analysis-based test tagging

Abstract

A test corpus is tagged based on the fault analysis. Tags associated with test cases known to execute without error relative to the original source code corpus are accessed. A test set associated with the accessed tag is accessed, the test set containing test cases. A new source code corpus to be tested is accessed. The new source code corpus is executed using the test cases from the accessed test set. An error generated by executing the new source code corpus using the test case is accessed. A location in a new source code corpus associated with the accessed error is determined. Natural language processing software is executed to generate a new marker associated with a location in a new source code corpus associated with the accessed error.

Description

Technical Field

[0001] The present invention relates generally to software development and, more particularly, to marking software tests based on failure analysis. Background Art

[0002] The presently disclosed embodiments relate to software development, and in particular, complex software development that occurs in an interactive development environment, such as that accessed by multiple individuals. In 2023, software development remains an important field, as software is increasingly complex and is written by a large number of developers working together on projects. However, errors and defects continue to be an obstacle to complex software development, as any developer knows that a small defect somewhere in the source code corpus may result in compilation errors, runtime errors, logical errors, incorrect results, etc. Developers may have to spend a lot of time correcting these errors.

[0003] In order to facilitate easier editing, runtime testing, verification, debugging of source code, etc., "tags" may be used with reference to a test corpus. Tags may, for example, identify one or more concepts, relationships, features, or resources that are confirmed / driven / executed by tests for the software or source code. Although tags can aid the source code development and testing process, tagging a test corpus is time consuming. As tests or source code are revised, as each version is verified, it may even be necessary to continue to add, edit, or modify tags for new versions of the tests or source code, and even greater amounts of effort must be spent each time a new corpus of source code or tests is edited, updated, revised, etc.

[0004] A systematic and continuous revalidation approach is needed to label the test corpus. Summary of the invention

[0005] Embodiments of the present invention disclose a method, system, and computer program product for labeling a test corpus based on fault analysis using a computing device. The computing device accesses one or more tags associated with one or more test cases that are known to be executed without errors relative to an original source code corpus. The computing device accesses a test set associated with one or more accessed tags, the test set containing one or more test cases. The computing device accesses a new source code corpus to be tested. The computing device executes the new source code corpus using one or more test cases from the accessed test set. One or more errors generated by executing the new source code corpus using the one or more test cases are accessed. One or more locations in the new source code corpus associated with the accessed one or more errors are determined. Natural language processing software is executed to generate one or more new tags associated with one or more locations in the new source code corpus associated with the accessed one or more errors.

[0006] In an alternative aspect of the present invention, an embodiment discloses a method, system, and computer program product for labeling a test corpus based on fault analysis using a computing device. The computing device accesses one or more tags associated with one or more test cases known to be executed without error relative to an original source code corpus. The computing device accesses a test set associated with one or more of the accessed tags, the test set containing one or more test cases. The computing device accesses a new source code corpus to be tested. The computing device executes the new source code corpus using one or more test cases from the accessed test set. One or more errors generated by executing the new source code corpus using the one or more test cases are accessed. One or more locations in the new source code corpus associated with the accessed one or more errors are determined. Natural language processing software is executed to generate one or more new tags associated with one or more locations in the new source code corpus associated with the accessed one or more errors. The computing device accesses the generated one or more new tags associated with one or more locations in the new source code corpus. The computing device accesses a new test set associated with the generated one or more new tags, the new test set providing one or more new test cases, the new test cases providing a wider range of tests on the new source code corpus associated with categories of similar errors.

[0007] In another optional aspect of the present invention, an embodiment discloses a method, system, and computer program product for labeling a test corpus based on fault analysis using a computing device. The computing device accesses one or more tags associated with one or more test cases that are known to be executed without errors relative to the original source code corpus. The computing device accesses a test set associated with one or more accessed tags, the test set containing one or more test cases. The computing device accesses a new source code corpus to be tested. The computing device executes the new source code corpus using one or more test cases from the accessed test set. One or more errors generated by executing the new source code corpus using the one or more test cases are accessed. One or more locations in the new source code corpus associated with the accessed one or more errors are determined. Natural language processing software is executed to generate one or more new tags associated with one or more locations in the new source code corpus associated with the accessed one or more errors. An artificial intelligence model associated with the natural language processing model is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A networked computer environment 100 is shown in accordance with an embodiment of the present invention.

[0009] Figure 2 is a functional block diagram illustrating a module 200 for labeling a test corpus based on fault analysis according to an embodiment of the present invention.

[0010] Figure 3 is a flow chart illustrating operational steps that may be performed by hardware components of a hardware device according to an embodiment of the present invention.

[0011] Figure 4A and Figure 4B is a flow chart illustrating operational steps that may be performed by hardware components of a hardware device according to an alternative embodiment of the present invention.

[0012] Figure 5 is a flow chart illustrating operational steps that may be performed by hardware components of a hardware device according to an alternative embodiment of the present invention. DETAILED DESCRIPTION

[0013] The presently disclosed embodiments relate to one or more methods, systems, and computer program products for marking a software test corpus based on fault analysis. Systematic and continuous revalidation of test marking facilitates many tasks that may be involved in software development, including source code verification, source code debugging, cluster testing, and the like. Natural language processing software, such as supported by artificial intelligence software including large language models, pipelines the task of marking tests in an automated manner. Embodiments of the present invention rely on or build upon previous executions of embodiments, marking tests using regression analysis of results from previous executions, where the results from previous executions are refined and further detailed. The regression analysis can be extended to previous versions of source code, rollouts, and the like, or other source code corpora that may be related in some way, including updates to tests that verify, drive, or execute source code.

[0014] The presently disclosed embodiments may be used in conjunction with any environment in which source code is being developed, tested, accessed, edited, modified, etc., such as within an interactive development environment that allows multiple developers, project managers, etc. to develop source code (and provides functionality such as viewing, editing, compiling, executing, testing, and debugging source code). Embodiments of the present invention streamline the software development process by allowing individuals involved in the development process to systematically tag their test corpora. In general, embodiments of the present invention may be implemented in different ways while being contemplated by embodiments of the present invention. The presently disclosed embodiments may, for example, be implemented as a plug-in to an interactive development environment, a stand-alone application, as part of a web browser, or in any other way while being contemplated by embodiments of the present invention disclosed herein.

[0015] Various aspects of the present disclosure are described by narrative text, flow charts, block diagrams of computer systems, and / or block diagrams of machine logic included in computer program product (CPP) embodiments. With respect to any flow chart, depending on the technology involved, the operations may be performed in an order different from the order shown in a given flow chart. For example, again depending on the technology involved, two operations shown in consecutive flow chart blocks may be performed in reverse order, as a single integrated step, simultaneously, or in a manner that at least partially overlaps in time.

[0016] Computer program product embodiments ("CPP embodiments" or "CPP") are terms used in this disclosure to describe any collection of one or more storage media (also referred to as "media") collectively included in a collection of one or more storage devices that collectively include machine-readable code corresponding to instructions and / or data for performing the computer operations specified in a given CPP claim. A "storage device" is any tangible device that can hold and store instructions for use by a computer processor. Without limitation, a computer-readable storage medium may be an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, a mechanical storage medium, or any suitable combination of the foregoing. Some known types of storage devices that include these media include: magnetic disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), static random access memories (SRAM), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, mechanical encoding devices (such as punch cards or pits / land formed in a major surface of a disk), or any suitable combination of the foregoing. Computer-readable storage media, as the term is used in this disclosure, should not be construed as storing in the form of transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides, light pulses through fiber optic cables, electrical signals transmitted through wires, and / or other transmission media. As will be appreciated by those skilled in the art, data is typically moved at certain occasional points in time during normal operation of the storage device, such as during access, defragmentation, or garbage collection, but this does not make the storage device transitory because the data is not transitory while it is stored.

[0017] The computing environment 100 includes an example of an environment for executing at least some of the computer codes involved in performing the methods of the present invention, such as associated with a module 200 for labeling a test corpus based on fault analysis. In addition to the module 200, the computing environment 100 includes, for example, a computer 101, a wide area network (WAN) 102, an end-user device (EUD) 103, a remote server 104, a public cloud 105, and a private cloud 106. In this embodiment, the computer 101 includes a processor set 110 (including a processing circuit 120 and a cache 121), a communication structure 111, a volatile memory 112, a persistent storage device 113 (including an operating system 122 and the module 200, as described above), a peripheral device set 114 (including a user interface (UI) device set 123, a storage device 124, and an Internet of Things (IoT) sensor set 125), and a network module 115. The remote server 104 includes a remote database 130. The public cloud 105 includes a gateway 140 , a cloud orchestration module 141 , a host physical machine set 142 , a virtual machine set 143 , and a container set 144 .

[0018] Computer 101 may take the form of a desktop computer, laptop computer, tablet computer, smart phone, smart watch or other wearable computer, mainframe computer, quantum computer, or any other form of computer or mobile device now known or developed in the future that is capable of running programs, accessing a network, or querying a database such as remote database 130. As is well known in the art of computer technology, and depending on the technology, the performance of computer-implemented methods may be distributed among multiple computers and / or among multiple locations. On the other hand, in this presentation of computing environment 100, the detailed discussion focuses on a single computer, particularly computer 101, to keep the presentation as simple as possible. Computer 101 may be located in the cloud, even though Figure 1 1 is not shown in the cloud, on the other hand, computer 101 need not be in the cloud except to any extent that can be positively indicated.

[0019] The processor set 110 includes one or more computer processors of any type now known or developed in the future. The processor set 110 may be referred to here alternatively as one or more "computing devices", but the computing device may also refer to one or more CPUs, microchips, integrated circuits, embedded systems, or equivalents that exist currently or appear later. The processing circuit 120 may be distributed over multiple packages, such as multiple coordinated integrated circuit chips. The processing circuit 120 may implement multiple processor threads and / or multiple processor cores. The cache 121 is a memory located in the processor chip package and is generally used for data or code that should be available for fast access by threads or cores running on the processor set 110. Cache memories are generally organized into multiple levels according to relative proximity to the processing circuit. Alternatively, some or all of the caches of the processor set may be located "off-chip". In some computing environments, the processor set 110 may be designed to work with qubits and perform quantum computing.

[0020] Computer-readable program instructions are typically loaded onto the computer 101 to cause the processor set 110 of the computer 101 to perform a series of operating steps to implement a computer-implemented method, such that the instructions so executed will instantiate the method specified in the flowchart and / or the narrative description of the computer-implemented method included in this document (collectively referred to as the "method of the present invention"). These computer-readable program instructions are stored in various types of computer-readable storage media, such as cache 121 and other storage media discussed below. The program instructions and related data are accessed by the processor set 110 to control and direct the execution of the method of the present invention. In the computing environment 100, at least some of the instructions for performing the method of the present invention may be stored in a module 200 in a persistent storage device 113.

[0021] The communication fabric 111 is the signal conduction paths that allow the various components of the computer 101 to communicate with each other. Typically, the fabric is made up of switches and conductive paths, such as those that make up a bus, a bridge, physical input / output ports, etc. Other types of signal communication paths may be used, such as fiber optic communication paths and / or wireless communication paths.

[0022] The volatile memory 112 is any type of volatile memory now known or developed in the future. Examples include dynamic random access memory (RAM) or static RAM. Typically, the volatile memory 112 is characterized as random access, but this is not required unless expressly stated. In the computer 101, the volatile memory 112 is located in a single package and is internal to the computer 101, but, alternatively or additionally, the volatile memory may be distributed among multiple packages and / or located externally relative to the computer 101.

[0023] Persistent storage 113 is any form of non-volatile storage for computers known now or developed in the future. The non-volatility of the storage means that the stored data is maintained regardless of whether power is supplied to the computer 101 and / or directly to the persistent storage 113. Persistent storage 113 may be a read-only memory (ROM), but typically at least a portion of the persistent storage allows writing of data, deletion of data, and rewriting of data. Some common forms of persistent storage include magnetic disks and solid-state storage devices. Operating system 122 may take several forms, such as various known proprietary operating systems or operating systems of the open source portable operating system interface type using a kernel. The code included in module 200 typically includes at least some of the computer code involved in executing the method of the present invention.

[0024] The peripheral device set 114 includes a peripheral device set of the computer 101. The data communication connection between the peripheral device and other components of the computer 101 can be implemented in various ways, such as a Bluetooth connection, a near field communication (NFC) connection, a connection made by a cable (such as a universal serial bus (USB) type cable), a plug-in type connection (e.g., a secure digital (SD) card), a connection made through a local area communication network, and even a connection made through a wide area network such as the Internet. In various embodiments, the UI device set 123 may include components such as a display screen, a speaker, a microphone, a wearable device (such as goggles and smart watches), a keyboard, a mouse, a printer, a touchpad, a game controller, and a tactile device. The storage device 124 is an external storage device, such as an external hard drive, or a pluggable storage device, such as an SD card. The storage device 124 can be persistent and / or volatile. In some embodiments, the storage device 124 can take the form of a quantum computing storage device for storing data in the form of quantum bits. In embodiments where the computer 101 needs to have a large amount of storage (e.g., where the computer 101 locally stores and manages a large database), the storage may be provided by a peripheral storage device designed to store very large amounts of data, such as a storage area network (SAN) shared by multiple geographically distributed computers. The IoT sensor set 125 is made up of sensors that can be used in IoT applications. For example, one sensor may be a thermometer, while another sensor may be a motion detector.

[0025] The network module 115 is a collection of computer software, hardware, and firmware that allows the computer 101 to communicate with other computers via the WAN 102. The network module 115 may include hardware such as a modem or a Wi-Fi signal transceiver, software for packetizing and / or depacketizing data transmitted over a communication network, and / or web browser software for transmitting data over the Internet. In some embodiments, the network control function and the network forwarding function of the network module 115 are executed on the same physical hardware device. In other embodiments (e.g., embodiments utilizing software defined networks (SDN)), the control function and the forwarding function of the network module 115 are executed on physically separated devices, so that the control function manages several different network hardware devices. Computer-readable program instructions for executing the method of the present invention can typically be downloaded to the computer 101 from an external computer or an external storage device via a network adapter card or a network interface included in the network module 115.

[0026] WAN 102 is any wide area network (e.g., the Internet) capable of transmitting computer data over non-local distances by any technology now known or developed in the future for transmitting computer data. In some embodiments, WAN 102 may be replaced and / or supplemented by a local area network (LAN) designed to transmit data between devices located in a local area, such as a Wi-Fi network. WANs and / or LANs typically include computer hardware, such as copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and edge servers.

[0027] End-user device (EUD) 103 is any computer system used and controlled by an end-user (e.g., a customer of an enterprise operating computer 101), and may take any of the forms discussed above in connection with computer 101. EUD 103 typically receives helpful and useful data from the operation of computer 101. For example, in the hypothetical case where computer 101 is designed to provide recommendations to an end-user, the recommendations would typically be transmitted from network module 115 of computer 101 to EUD 103 via WAN 102. In this manner, EUD 103 may display or otherwise present the recommendations to the end-user. In some embodiments, EUD 103 may be a client device, such as a thin client, a fat client, a mainframe computer, a desktop computer, or the like.

[0028] Remote server 104 is any computer system that provides at least some data and / or functionality to computer 101. Remote server 104 may be controlled and used by the same entity that operates computer 101. Remote server 104 represents a machine that collects and stores helpful and useful data for use by other computers, such as computer 101. For example, in the hypothetical case where computer 101 is designed and programmed to provide recommendations based on historical data, then the historical data may be provided to computer 101 from remote database 130 of remote server 104.

[0029] The public cloud 105 is any computer system that can be used by multiple entities, which provides on-demand availability of computer system resources and / or other computer capabilities, particularly data storage (cloud storage) and computing capabilities, without the need for direct active management by users. Cloud computing typically uses the sharing of resources to achieve consistency and economy of scale. The direct and active management of the computing resources of the public cloud 105 is performed by the computer hardware and / or software of the cloud orchestration module 141. The computing resources provided by the public cloud 105 are typically implemented by virtual computing environments running on various computers that constitute the host physical machine set 142, which is a full domain of physical computers in the public cloud 105 and / or available for the public cloud. The virtual computing environment (VCE) is typically in the form of a virtual machine from the virtual machine set 143 and / or a container from the container set 144. It should be understood that these VCEs can be stored as images and can be transferred between various physical machine hosts as images or after the instantiation of the VCE. The cloud orchestration module 141 manages the transfer and storage of images, deploys new instantiations of VCEs, and manages active instantiations of VCE deployments. Gateway 140 is a collection of computer software, hardware, and firmware that allows public cloud 105 to communicate over WAN 102 .

[0030] Some further explanation of virtualized computing environments (VCEs) will now be provided. A VCE can be stored as an "image" and new active instances of the VCE can be instantiated from the image. Two common types of VCEs are virtual machines and containers. Containers are VCEs that use operating system-level virtualization. This refers to an operating system feature in which the kernel allows the existence of multiple isolated user space instances, called containers. From the perspective of the programs running in them, these isolated user space instances generally behave like actual computers. Computer programs running on a normal operating system can utilize all of the resources of that computer, such as connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, programs running within a container can only use the contents of the container and the devices assigned to the container, a feature known as containerization.

[0031] Private cloud 106 is similar to public cloud 105, except that the computing resources are only available to a single enterprise. Although private cloud 106 is depicted as communicating with WAN 102, in other embodiments, the private cloud can be completely disconnected from the Internet and only accessible through a local / private network. A hybrid cloud is a combination of multiple clouds of different types (e.g., private, community, or public cloud types), typically implemented by different vendors. Each of the multiple clouds remains a separate and discrete entity, but the larger hybrid cloud architecture is bound together by standardized or proprietary technologies that enable orchestration, management, and / or data / application portability between the multiple constituent clouds. In this embodiment, both public cloud 105 and private cloud 106 are part of a larger hybrid cloud.

[0032] Figure 2 is a functional block diagram illustrating a module 200 for labeling a test corpus based on fault analysis according to an embodiment of the present invention. As an overview, Figure 2 An interactive development environment 210, a marking module 260, and a network 299 are shown. The interactive development environment 210 allows individuals involved in the source code development process to perform various functions related to the source code, including accessing, storing, editing, compiling / interpreting, testing, and debugging different versions of the source code. Embodiments of the present invention can be used in conjunction with various types of source code, including compiled, interpreted scripts, bytecodes, or even previously developed software. All of these require testing and can benefit from automatic marking, as discussed in embodiments of the present invention. The various functions provided by the interactive development environment 210 can allow developers, project managers, or other users interested in the development process to register, log out, store, edit, compile / interpret, execute, and debug source code, as well as perform other functions in conjunction with different versions of the source code stored by the interactive development environment 210. The interactive development environment 210 actually provides one or more users with an interface for performing various functions related to source code and / or software. The interactive development environment 210 can be implemented as any type of computer software (and in various embodiments, associated computer hardware) that performs functions as discussed herein. Interactive development environment 210 operates in conjunction with tagging module 260 to systematically and continuously tag a test corpus, as discussed further herein.

[0033] Also there Figure 2The tagging module 260 shown in FIG. 2 is operatively connected to the interactive development environment 210 directly or via the network 299. Alternatively, the tagging module 260 is integrated with the interactive development environment 210, such as via a plug-in to the interactive development environment 210, access to / via a public web browser, or other equivalent means. Typically, and as further discussed in conjunction with the embodiments of the invention disclosed herein, the tagging module 260 is responsible for tagging the test corpus in an automated manner, and in various embodiments of the invention, natural language processing software or other techniques are used to systematically generate tags for various versions of the test. As will be appreciated by those skilled in the art, "tags" may be used in various ways to indicate revisions, different versions, degrees of change, comments, etc., in various different corpora of source code available from the interactive development environment 210. Developers may utilize tags, for example, to help manually discover vulnerabilities in source code, generate cluster tests, etc., or to help automatically debug source code. In embodiments of the invention, as further discussed herein, the tagging module 260 operates via regression to leverage results of previous executions of the tagging module 260 in order to obtain optimal results (i.e., obtain the "optimal" tagging for the test corpus) in an accelerated and efficient manner. The tagging module 260 may be implemented as any kind of computer hardware (and, in various embodiments, associated computer hardware for performing the functions discussed). In embodiments of the invention, the tagging module 260 may, for example, access source code available from the interactive development environment 210, access features of the interactive development environment 210 to automatically compile, interpret, or otherwise execute source code, and, as further discussed herein, systematically tag tests accessed from the interactive development environment 210.

[0034] like Figure 2 As further shown in FIG. 1 , in various embodiments of the present invention, the tagging module 260 and the interactive development environment 210 are connected via a network 299. In various embodiments of the present invention, the network 299 represents, for example, any type of computer network, such as a local area network (LAN), a wide area network (WAN) such as the Internet, and includes wired, wireless, or fiber optic connections. In various embodiments, the network 299 is substantially the same as the WAN 102, as described herein in conjunction with FIG. Figure 1 Generally, according to embodiments of the present invention, network 299 may be any combination of connections and protocols that will support communications between interactive development environment 210 and marking module 260. In further embodiments of the present invention, network 299 may represent an internal bus associated with a single or multi-core processor executing both interactive development environment 210 and marking module 260 (such as in embodiments where interactive development environment 210 and marking module 260 are integrated).

[0035] Discuss in more detail Figure 2, the interactive development environment 210 represents computer software (and, in various embodiments, associated computer hardware) for performing various functions related to source code, including accessing, storing, editing, compiling / interpreting, executing, and debugging a corpus of source code. As will be appreciated by those skilled in the art, the interactive development environment 210 provides various tools necessary for one or more developers, project managers, etc. to develop, test, execute, debug software, etc. As described elsewhere herein, in embodiments of the present invention, the interactive development environment 210 is operably connected to, or integrated with, the tagging module 260.

[0036] In various embodiments of the invention, interactive development environment 210 includes one or more of source code repository 213 , source code editor 215 , build automation 218 , and debugger 223 .

[0037] Source code repository 213 represents software and / or hardware for storing, accessing, checking in, checking out, saving, etc. different corpora of source code available from developers (such as via uploading, via source code editor 215 or otherwise). Source code repository 213 stores source code in an accessible form. As will be appreciated by those skilled in the art, as multiple developers, project managers, etc. continue to edit source code, it is important to store multiple versions of the uploaded source code corpus in case it is necessary to roll back to a different version (such as due to the loss of important parts, vulnerabilities, poorly designed new objects, etc.). If a downstream developer makes a big mistake, it is a simple matter to simply roll back to a previous version using source code repository 213 instead of rewriting the entire source code corpus. However, as discussed in the context of the disclosed embodiments, when a new corpus, version, etc. of source code is registered in a source code repository, it may be necessary, such as via concepts, execution, and / or logic, to "mark" important tests associated with the source code. In conjunction with embodiments of the present invention, the time-consuming task of marking an existing or new test corpus of source code in repository 213 can be pipelined. In various embodiments of the invention, source code repository 213 stores one or more test sets associated with each source code corpus (each containing one or more test cases for testing the source code corpus) and one or more updates to the test sets (as discussed further herein).

[0038] Source code editor 215 represents software and / or hardware for providing users, developers, project managers, etc. with tools for developing and editing source code in interactive development environment 210. Source code editor 215 may also provide developers, project managers, etc. with various options to check in source code, request compilation of source code, execute source code, debug source code, etc., as discussed in conjunction with source code repository 213, source code editor 215, build automation 218, and debugger 223.

[0039] Build automation 218 represents software and / or hardware for automatically (or upon request) compiling a source code corpus stored in source code repository 213. Depending on the type of source code under consideration, build automation 218 may be associated with a compiler, an interpreter, or in another equivalent manner to convert high-level, human-readable source code into machine language for execution by a computing device. In the context of an embodiment of the present invention, build automation 218 may be called by a tagging module 260 to request compilation and / or execution of a source code corpus for further use (such as for debugging or automatically tagging a test corpus) as discussed in the embodiments disclosed herein. In various embodiments of the present invention, build automation 218 is associated with one or more compilers / debuggers.

[0040] Debugger 223 represents software and / or hardware for automatically debugging software or helping developers to debug manually. Debugger 223 can be associated with build automation 218 and / or compiler / interpreter (not shown). In the context of compiling, interpreting or otherwise executing a source code corpus, the debugger can track one or more errors, including, for example, compilation errors, runtime errors, or other errors generated by a compiler / interpreter (not shown, associated with build automation 218). Errors tracked by debugger 223 can be sent to a marking module 260 for further use. In the context of embodiments of the present invention, tracked errors (as discussed more fully herein) can be utilized in various ways.

[0041] Continue to refer Figure 2Element, marking module 260 represents computer software (and in various embodiments, associated computer hardware) for marking test corpora in an automated manner. As described elsewhere herein, source code corpora are maintained by interactive development environment 210, including different versions, updates, edits, etc. of related source code projects. As understood by those skilled in the art, interactive development environment 210 stores different versions of source code, etc., because it may be necessary to roll back to a previous version, such as if the current version of the source code is wrong, does not provide correct results, etc. In general, marking module 260 presents a technique for systematically marking test corpora associated with different versions of source code, and creates improvements and generates new tags based on previous tags via regression execution, as further discussed herein. New tags generated by marking module 260 can be displayed to the user via a user interface associated with interactive development environment 210, such as source code editor 215.

[0042] In various embodiments of the present invention, the marking module 260 includes one or more of a marking database 262, a test case module 265, a source code access and execution module 268, an error module 271, a natural language marking module 276, a marking database update module 278, a test execution history 279, a user access module 280, and a machine learning improvement module 285.

[0043] The tag database 262 represents software and / or hardware for storing, accessing, and making available one or more "tags" associated with different test versions available from the interactive development environment 210. In various embodiments, the tags stored by the tag database 262 may be associated with one or more test cases that are known to execute without error relative to the original source code corpus available from the interactive development environment 210 or generated by other iterations of embodiments of the present invention. In embodiments of the present invention, execution begins with accessing, by a computing device, from the tag database 262, one or more tags associated with one or more test cases that are known to execute without error relative to the original source code corpus. Starting execution with tags associated with test cases that are known to execute without error relative to the original source code corpus provides efficient and effective tagging of the relevant test corpus, wherein the relevant "tags" are used as starting points for executing other aspects or iterations of the present invention. In embodiments of the present invention, the tag database 262 is also responsible for storing one or more newly generated tags, which are further used in regression execution of embodiments of the present invention to generate updated or more detailed tags for other test corpora. As new tags are generated through iterations of embodiments of the present invention, they are stored by the tag database 262 for further utilization, as discussed further herein.

[0044] The test case module 265 represents software and / or hardware for accessing, downloading, modifying, updating, uploading, etc., one or more test sets from the interactive development environment 210. As discussed in conjunction with the tag database 262 and other embodiments disclosed herein, a test set is associated with one or more accessed tags. Each test set accessed from the interactive development environment 210 may contain one or more test cases for testing a source code corpus, including, for example, test variables, methods, interfaces, inputs, outputs, etc., and in an embodiment of the present invention, test concepts, relationships, features, resources, which can be confirmed / driven / executed by the test. In an embodiment of the present invention, when execution begins, the test set accessed from the test case module 265 is known to contain one or more known test cases that execute without error relative to the original source code corpus accessed from the interactive development environment 210. In various embodiments of the present invention, the test case module 265 also provides the function of accessing a new test set of one or more new test cases, the new test set being associated with one or more generated new tags (as further discussed herein). As discussed herein, a "new test set" refers to an update, change, or modification from a "test case known to execute without error." The new test set associated with the generated one or more new tags provides more extensive testing of the new source code corpus (i.e., the new test set consisting of one or more test cases can provide more detailed information about the types of errors encountered in debugging because the new test set has been previously associated with other corpora / subjects of source code that had similar errors in the past. The other corpora / subjects can be new source code corpora, completely unrelated source code, previous versions of compiled software, or some combination of these). The new test set can be automatically derived from other corpora / subjects of source code (e.g., by machine learning models), manually created, etc. The errors encountered by the previous version of the source code and the associated tests will likely be similar to the new errors encountered by the newer version of the source code and the associated tests, so it is a good starting point to execute the new embodiment of the present invention with the newer version of the source code and the associated tests. As the execution of the embodiments of the present invention proceeds, the test case module 265 can also be associated with the generation of updated test sets, which are associated with one or more new errors available from the error module 271, etc. In conjunction with these embodiments, new locations in the new source code corpus are associated with the one or more new errors accessed and the one or more updated tags. As will be appreciated by those skilled in the art, continued execution of embodiments of the present invention produces more sophisticated and detailed results via regression execution. In various embodiments of the present invention, one or more newly generated new tags may also be associated with the test execution history 279 to provide further information about executing different versions of source code, errors, tags, etc. to developers, project managers, etc.

[0045] Source code access and execution module 268 represents software and / or hardware for accessing source code corpora from interactive development environment 210. Source code stored by interactive development environment 210 may be in different forms, such as uncompiled, or partially compiled, and may be of different types, such as interpreted, scripted, etc. As developers continue to edit, modify, and upload new source code corpora to interactive development environment 210, the interactive development environment continues to store (also referred to as "check-in") previous versions of the source code, because it may be necessary to "roll back" to a previous version, or for other reasons. As discussed herein, different versions of the source code are used. For clarity, "original source code corpus" refers to the first version of such source code under consideration, while "new source code corpus" refers to another source code corpus containing edits, improvements, modifications, etc. to the "original source code corpus". In various embodiments of the present invention, newer and updated edits, improvements, modifications, etc. of each successive version may be stored, accessed, marked, etc. by interactive development environment 210. Through the recursive execution of embodiments of the present invention, newer and updated versions of the source code may be accessed and modified in conjunction with further iterations of the present invention. In an alternative embodiment of the invention, the new source code corpus is simply completely new code (relative to the original source code corpus).Performing embodiments of the invention in conjunction with newer and updated versions of source code is used to update a test corpus to reflect newer and updated versions of source code, etc.

[0046] The source code access and execution module 268 is also used to request the interactive development environment 210 to execute various corpora, versions, etc. of the source code. The execution requested by the source code access and execution module 268 is related to the execution of a specific corpus of the source code using a certain test case set from a certain test set. As discussed herein, various test case sets may be referred to as "one or more test cases known to be executed without error", "new test sets", "updated test sets", etc. When the various embodiments of the present invention are executed, it is discussed that the test cases are executed via regression execution operations as further discussed herein for new markings regarding updates to the test corpus. As execution proceeds, execution may be requested with new test cases from new test sets or updated test sets, etc. As understood by those skilled in the art, each successive test set is an update or newly generated iteration of the previous test set so as to mark the test corpus in an efficient manner.

[0047] The error module 271 represents software and / or hardware for accessing one or more errors associated with the execution of various corpora of source code with different test sets. After being requested by the source code access and execution module 268 to execute a specific source code corpus with a specific test set, the interactive development environment 210 accesses one or more errors generated by the execution of the specific source code corpus. These errors may include, but are not limited to, compilation errors, runtime errors, logical errors, incorrect results, etc. When one or more errors are generated by executing various corpora of source code, these errors are accessed by the error module 271 for further use as discussed herein. Also as discussed herein, the error module 271 is also associated with the detection of new errors detected by executing the generation of a new source code corpus using one or more new test cases, the detection of further errors associated with executing various source code corpora, etc. Typically, an error may be associated with one or more newly edited sections of different versions of the source code relative to a previous version (such as because, for example, a developer mistypes a method or variable name in a new section of the source code).

[0048] The error module 271 and the source code access and execution module 268 and the Figure 2 , is used to determine one or more locations in various source code corpora associated with the accessed errors in combination with other modules discussed in . Continuing with the previous example, if the developer mistypes the variable "foo", resulting in a runtime error, the error module 271 locates where the mistyped variable "foo" is in the source code corpus. As execution proceeds, the error module 271 can similarly determine one or more new locations in new source code corpora associated with new errors, and so on. In various embodiments of the present invention, the error module 271 can simply receive the location of the error in the source code corpus from the interactive development 210, as well as which part of the source code received the error, and what variable / method or equivalent means.

[0049] The natural language tagging module 276 represents software and / or hardware for generating new tags for various test corpora. After accessing (or otherwise detecting) errors and determining one or more locations associated with the errors by the error module 271, the natural language tagging module 276 automatically tags these potential error areas of the source code corpus, and in an embodiment of the present invention, also tags one or more tests in the test corpus associated with the errors. In various embodiments of the present invention, when tagging the test corpus, the natural language tagging module 276 can rely at least in part on various development metadata, such as development metadata associated with the source code development process, preferred edits made by different developers, etc.

[0050] In various embodiments of the present invention, the natural language tagging module 276 can rely on an artificial intelligence-based model (e.g., a large language model) to easily and quickly generate new tags for an associated test corpus. The large language model can be trained, retrained, etc. using previously labeled source code corpora and test sets to quickly generate new tags in an easy manner. The large language model used can be based on general utilization of source code, but is improved by training, retraining, etc. through a specially labeled similar test corpus (as further discussed in conjunction with the machine learning improvement module 285). In various embodiments of the present invention, as regression execution continues, the natural language tagging module 276 continues to generate updated tags for further downstream versions of the test, wherein the updated tags provide more detailed fault analysis of new source code corpora available from the interactive development environment 210, etc. (as further discussed herein). "New tags", "newer tags", "updated tags", etc. are submitted to the tag database 262 when they are created for further use in future iterations of embodiments of the present invention.

[0051] The tag database update module 278 represents software and / or hardware for updating the record of tags generated in conjunction with the iterative regression execution of the present invention. As the execution continues (as discussed herein), the record of tags is utilized to make more detailed tags.

[0052] Test execution history 279 represents software and / or hardware for storing records of execution of previous iterations of the disclosed embodiments. In various embodiments, test execution history 279 contains records of different source code, associated test corpora (i.e., test cases), associated errors, and other data discussed herein to be used in embodiments of the present invention.

[0053] The user access module 280 represents software and / or hardware for a user to access and view one or more new tags, a test corpus, one or more locations determined in a new source code corpus associated with the accessed one or more errors, and other data discussed in conjunction with embodiments of the present invention. Various users (developers, project managers, etc.) can access the one or more errors via the interactive development environment 210 (in communication with the user access module 280) or in another manner.

[0054] The machine learning improvement module 285 represents software and / or hardware for improving the operation of a machine learning module used in conjunction with an embodiment of the present invention (e.g., in conjunction with the natural language tagging module 276). In embodiments where the natural language tagging module 276 utilizes a large language model, for example, the large language model may have been previously trained primarily on a corpus of novels and a small amount of source code, so the machine learning improvement module 285 can specifically utilize a corpus of source code, test cases, etc. to "improve" the large language model for better utilization in embodiments of the present invention. For example, a corpus of labeled test cases can be used, for example, to train or retrain a large language model. In embodiments of the present invention, where different types of artificial intelligence, neural networks, etc. are used in conjunction with natural language processing software (discussed in conjunction with the natural language tagging module 276), the machine learning improvement module 285 can be used to improve the functionality of those.

[0055] Figure 3 Flowcharts depicting operational steps that may be performed by a hardware component, multiple hardware components, and / or hardware devices according to embodiments of the present invention. Figure 3 As shown, at step 310, the tag database 262 accesses one or more tags associated with one or more test cases that are known to execute without errors relative to the original source code corpus. At step 320, the test case module 265 accesses a test set associated with one or more of the accessed tags, the test set containing one or more test cases. At step 330, the source code access and execution 218 module accesses a new source code corpus to be tested. At step 340, the source code access and execution module 268 executes the new source code corpus using one or more test cases from the accessed test set. At step 350, the error module 271 accesses one or more errors generated by executing the new source code corpus using the one or more test cases. At step 360, the error module 271 determines one or more locations in the new source code corpus associated with the accessed one or more errors. At step 370, the natural language tagging module 276 executes natural language processing software to generate one or more new tags associated with one or more locations in the new source code corpus associated with the accessed one or more errors. Such as Figure 3 Implementation of the embodiments presented in the present invention presents the advantages of systematically "tagging" new test corpora to present time savings, productivity, and accuracy regarding these automatically generated tags to developers, project managers, etc., and anyone else interested in source code or other software.

[0056] Figure 4A and Figure 4B 1 is a flow chart illustrating alternative operational steps that may be performed by a hardware component, multiple hardware components, and / or hardware devices according to an embodiment of the present invention. Figure 4A As shown, at step 410, the tag database 262 accesses one or more tags associated with one or more test cases that are known to be executed without errors relative to the original source code corpus. At step 420, the test case module 265 accesses a test set associated with one or more of the accessed tags, the test set containing one or more test cases. At step 430, the source code access and execution 218 module accesses a new source code corpus to be tested. At step 440, the source code access and execution module 268 executes the new source code corpus using one or more test cases from the accessed test set. At step 450, the error module 271 accesses one or more errors generated by executing the new source code corpus using the one or more test cases. At step 460, the error module 271 determines one or more locations in the new source code corpus associated with the accessed one or more errors. At step 470, the natural language tag module 276 executes natural language processing software to generate one or more new tags associated with one or more locations in the new source code corpus associated with the accessed one or more errors.

[0057] When execution continues to Figure 4B At step 480, the tag database 262 accesses the generated one or more new tags associated with one or more locations in the new source code corpus. At step 490, the test case module 265 accesses a new test set associated with the generated new tags, the new test set providing new test cases that provide more extensive testing of the new source code corpus associated with categories of similar errors. Figure 4A and Figure 4B Execution of the embodiments presented in the disclosure presents the advantage of using automatically generated new tags to automatically and more thoroughly tag a test corpus that may be more relevant to the type of automatically generated tags, presenting further time savings, work efficiency, and accuracy regarding these automatically generated tags to developers, project managers, etc., and anyone else interested in source code or other software, especially when new versions of source code or software are released.

[0058] Figure 5 is a flowchart depicting alternative operational steps that may be performed by a hardware component, multiple hardware components, and / or hardware devices according to embodiments of the present invention. Figure 5As shown, at step 510, the tag database 262 accesses one or more tags associated with one or more test cases that are known to be executed without errors relative to the original source code corpus. At step 520, the test case module 265 accesses a test set associated with one or more of the accessed tags, the test set containing one or more test cases. At step 530, the source code access and execution 218 module accesses a new source code corpus to be tested. At step 540, the source code access and execution module 268 executes the new source code corpus using one or more test cases from the accessed test set. At step 550, the error module 271 accesses one or more errors generated by executing the new source code corpus using one or more test cases. At step 560, the error module 271 determines one or more locations in the new source code corpus associated with the accessed one or more errors. At step 570, the natural language tag module 276 executes natural language processing software to generate one or more new tags associated with one or more locations in the new source code corpus associated with the accessed one or more errors. At step 580, the machine learning improvement module 285 improves the artificial intelligence model associated with the natural language processing module. Such as Figure 5 The execution of the embodiment presented in the present invention presents the advantages of generating new tags for a new source code corpus (saving time, improving work efficiency, and improving accuracy) and automatically improving the artificial intelligence model used in the manner discussed herein. The improvement of the artificial intelligence model is useful in conjunction with the execution of other embodiments of the present invention, for example to improve the accuracy and speed of prediction.

[0059] Based on the foregoing, methods, systems and computer program products have been disclosed. However, many modifications and substitutions may be made without departing from the scope of the present invention. Therefore, the present invention has been disclosed by way of example rather than limitation.

Claims

1. A method for labeling a test corpus based on fault analysis using a computing device, the method comprising: accessing, by a computing device, one or more tokens associated with one or more test cases known to execute without error relative to an original corpus of source code; Accessing, by the computing device, a test set associated with the one or more accessed tags, the test set comprising one or more test cases; accessing, by the computing device, a new source code corpus to be tested; executing, by the computing device, the new source code corpus using one or more test cases from the accessed test set; accessing one or more errors generated by executing the new source code corpus using the one or more test cases; determining one or more locations in the new source code corpus associated with the accessed one or more errors; as well as Natural language processing software is executed by the computing device to generate one or more new tags associated with the one or more locations in the new source code corpus associated with the accessed one or more errors.

2. The method according to claim 1, further comprising: accessing, by the computing device, the generated one or more new tokens associated with the one or more locations in the new source code corpus; as well as A new test set associated with the generated one or more new tags is accessed by the computing device, the new test set providing one or more new test cases, the one or more new test cases providing more extensive testing of the new source code corpus associated with categories of similar errors.

3. The method according to claim 2, further comprising: executing, by the computing device, the new source code corpus using one or more new test cases from the new test set; accessing one or more new errors generated by executing the new source code corpus using one or more new test cases; determining one or more new locations in the new source code corpus associated with the accessed one or more new errors; as well as Natural language processing software is executed by the computing device to generate one or more updated tags associated with the one or more new locations in the new source code corpus, the one or more new locations in the new source code corpus being associated with the one or more new errors accessed, the updated tags providing a more detailed fault analysis of the new source code corpus.

4. The method of claim 2, wherein the new test set is derived from other source code corpora and similar errors previously encountered.

5. The method of claim 3 , further comprising generating an updated test set associated with the one or more new errors, the new positions in the new source code corpus being associated with the accessed one or more new errors and the one or more updated tags.

6. The method of claim 1, wherein the new source code corpus to be tested is an update or new version of the original source code corpus.

7. The method of claim 6, wherein the one or more errors generated by executing the new source code corpus are associated with one or more edits in the new source code corpus relative to the original source code corpus.

8. The method of claim 1, wherein the original source code corpus is compiled or interpreted.

9. The method of claim 1, wherein the one or more new tokens and the determined one or more locations in the new source code corpus associated with the accessed one or more errors are displayed to one or more users via an interface.

10. The method of claim 1, wherein the new source code corpus, the one or more new tokens, and the determined one or more locations in the new source code corpus are used to improve an artificial intelligence model associated with the natural language processing software.

11. The method of claim 1 , wherein the new source code corpus is submitted to a repository.

12. The method of claim 11, wherein the repository contains source code, one or more test sets, and one or more updates to the test sets.

13. The method of claim 1 , wherein the one or more new tags associated with the one or more locations in the new source code corpus are generated based on a test execution history, the one or more locations in the new source code corpus being associated with the accessed one or more errors.

14. The method of claim 1, wherein the natural language processing software is associated with a large language model.

15. The method of claim 1, wherein the natural language processing software relies at least in part on development metadata when generating one or more new tags associated with the one or more locations in the new source code corpus.

16. A method of labeling a test corpus based on fault analysis using a computing device, the method comprising: accessing, by a computing device, one or more tokens associated with one or more test cases known to execute without error relative to an original corpus of source code; Accessing, by the computing device, a test set associated with the one or more accessed tags, the test set comprising one or more test cases; accessing, by the computing device, a new source code corpus to be tested; executing, by the computing device, the new source code corpus using one or more test cases from the accessed test set; accessing one or more errors generated by executing the new source code corpus using the one or more test cases; determining one or more locations in the new source code corpus associated with the accessed one or more errors; executing, by the computing device, natural language processing software to generate one or more new tags associated with the one or more locations in the new source code corpus associated with the accessed one or more errors; accessing, by the computing device, the generated one or more new tokens associated with the one or more locations in the new source code corpus; as well as A new test set associated with the generated one or more new tags is accessed by the computing device, the new test set providing one or more new test cases, the one or more new test cases providing more extensive testing of the new source code corpus associated with categories of similar errors.

17. The method according to claim 16, further comprising: executing, by the computing device, the new source code corpus using one or more new test cases from the new test set; accessing one or more new errors generated by executing the new source code corpus using the one or more new test cases; determining one or more new locations in the new source code corpus associated with the accessed one or more new errors; as well as Natural language processing software is executed by the computing device to generate one or more updated tags associated with the one or more new locations in the new source code corpus, the one or more new locations in the new source code corpus being associated with the accessed one or more new errors, the updated tags providing a more detailed fault analysis of the new source code corpus.

18. The method of claim 1, wherein the new test set is derived from other source code corpora and similar errors previously encountered.

19. A method of labeling a test corpus based on fault analysis using a computing device, the method comprising: accessing, by a computing device, one or more tokens associated with one or more test cases known to execute without error relative to an original corpus of source code; Accessing, by the computing device, a test set associated with the one or more accessed tags, the test set comprising one or more test cases; accessing, by the computing device, a new source code corpus to be tested; executing, by the computing device, the new source code corpus using one or more test cases from the accessed test set; accessing one or more errors generated by executing the new source code corpus using the one or more test cases; determining one or more locations in the new source code corpus associated with the accessed one or more errors; executing, by the computing device, natural language processing software to generate one or more new tags associated with the one or more locations in the new source code corpus associated with the accessed one or more errors; as well as An artificial intelligence model associated with the natural language processing model is improved by the computing device.

20. A computer system for labeling a test corpus based on fault analysis, the computer system comprising: one or more computer processors; one or more computer-readable storage media; Program instructions stored on the computer-readable storage medium for execution by at least one of the one or more processors, the program instructions comprising: program instructions for accessing one or more tokens associated with one or more test cases known to execute without error relative to an original corpus of source code; program instructions for accessing a test set associated with one or more accessed tags, the test set comprising one or more test cases; Program instructions for accessing a new corpus of source code to be tested; program instructions for executing the new source code corpus using one or more test cases from the accessed test set; program instructions for accessing one or more errors generated by executing the new source code corpus using the one or more test cases; program instructions for determining one or more locations in the new source code corpus associated with the accessed one or more errors; and Program instructions for executing natural language processing software to generate one or more new tags associated with the one or more locations in the new source code corpus associated with the accessed one or more errors.

21. The computer system of claim 20, further comprising: program instructions for accessing the generated one or more new tokens associated with the one or more locations in the new source code corpus; as well as Program instructions for accessing, by the computing device, a new test set associated with the generated one or more new tokens, the new test set providing one or more new test cases that provide more extensive testing of the new source code corpus associated with categories of similar errors.

22. The computer system of claim 20, wherein the new test set is derived from other source code corpora and similar errors previously encountered.

23. A computer program product for labeling a test corpus based on fault analysis, the computer program product comprising: One or more non-transitory computer-readable storage media and program instructions stored on the one or more non-transitory computer-readable storage media capable of executing a method comprising: accessing, by a computing device, one or more tokens associated with one or more test cases known to execute without error relative to an original corpus of source code; Accessing, by the computing device, a test set associated with the one or more accessed tags, the test set comprising one or more test cases; accessing, by the computing device, a new source code corpus to be tested; executing, by the computing device, the new source code corpus using one or more test cases from the accessed test set; accessing one or more errors generated by executing the new source code corpus using the one or more test cases; determining one or more locations in the new source code corpus associated with the accessed one or more errors; and executing, by the computing device, natural language processing software to generate one or more new tags associated with the one or more locations in the new source code corpus associated with the accessed one or more errors; and An artificial intelligence model associated with the natural language processing model is improved by the computing device.

24. The computer program product of claim 23, wherein the natural language processing software is associated with a machine learning model.

25. The computer program product of claim 24, wherein the machine learning model is a large language model.

26. A system comprising modules respectively configured to perform the steps of the method according to any one of claims 1 to 19.

Images