Test tagging based on failure analysis

Abstract

To provide tagging a corpus of tests based upon failure analysis.SOLUTION: Provided is tagging that includes accessing tags associated with test cases known to execute without error against an original corpus of source code and accessing a test set associated with the accessed tags, the test set containing the test cases. The tagging also includes accessing a new corpus of source code to be tested, executing the new corpus of the source code using test cases from the accessed test set, accessing errors generated by execution of the new corpus of source code using the test cases, determining locations in the new corpus of source code associated with the accessed errors, and executing natural language processing software to generate new tags associated with the locations in the new corpus of source code associated with the accessed errors.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates generally to software development, and more particularly to tagging software tests based on fault analysis. [Background technology]

[0002] FIELD OF THE DISCLOSURE The present disclosure relates to software development, and in particular to sophisticated software development, for example, performed in an interactive development environment accessed by multiple individuals. As of 2023, software development remains a critical field, as software is becoming increasingly complex and written by multiple developers collaborating on multiple projects. However, as every developer knows, errors and bugs continue to plague complex software development, as one small bug somewhere in the corpus of source code can lead to compilation errors, runtime errors, logic errors, inaccurate results, and the like. Developers can inevitably spend a great deal of time trying to correct these errors.

[0003] "Tags" may be used with reference to a corpus of tests to facilitate easier editing, runtime testing, validation, debugging, etc. of source code. Tags may, for example, identify one or more concepts, relationships, properties, or resources that are validated / driven / executed by tests on the software or source code. Although tags may aid in the source code development and testing process, tagging a corpus of tests is time consuming. As tests or source code are modified, it may further be necessary to continue adding, editing, or modifying tags for new versions of tests or source code as each version is validated, yet again requiring a significant amount of effort to be expended each time a new corpus of source code or tests is edited, updated, modified, etc.

[0004] The need for a systematic, continuously revalidated approach to tagging test corpora is demonstrated. Summary of the Invention

[0005] An embodiment of the present invention discloses a method, system, and computer program product for tagging a corpus of tests based on fault analysis using a computing device. The computing device accesses one or more tags associated with one or more test cases known to be executed without errors on an original corpus of source code. The computing device accesses a test set associated with the one or more accessed tags, the test set including one or more test cases. The computing device accesses a new corpus of source code to be tested. The computing device executes the new corpus of source code with one or more test cases from the accessed test set. One or more errors generated by execution of the new corpus of source code with the one or more test cases are accessed. One or more locations in the new corpus of source code associated with the accessed one or more errors are determined. Natural language processing software is executed to generate one or more new tags associated with the one or more locations in the new corpus of source code associated with the accessed one or more errors.

[0006] In an alternative aspect of the invention, embodiments disclose methods, systems and computer program products for tagging a corpus of tests based on fault analysis using a computing device. The computing device accesses one or more tags associated with one or more test cases known to be executed without errors on an original corpus of source code. The computing device accesses a test set associated with the one or more accessed tags, the test set including one or more test cases. The computing device accesses a new corpus of source code to be tested. The computing device executes the new corpus of source code with one or more test cases from the accessed test set. One or more errors generated by execution of the new corpus of source code with the one or more test cases are accessed. One or more locations in the new corpus of source code associated with the accessed one or more errors are determined. Natural language processing software is executed to generate one or more new tags associated with the one or more locations in the new corpus of source code associated with the accessed one or more errors. The computing device accesses the generated one or more new tags associated with the one or more locations in the new corpus of source code. The computing device accesses a new test set associated with the generated one or more new tags, the new test set providing one or more new test cases that provide more extensive testing of the new corpus of source code associated with similar categories of errors.

[0007] In another alternative aspect of the invention, embodiments disclose a method, system, and computer program product for tagging a corpus of tests based on fault analysis using a computing device. A computing device accesses one or more tags associated with one or more test cases known to be executed without errors on an original corpus of source code. The computing device accesses a test set associated with the one or more accessed tags, the test set including one or more test cases. The computing device accesses a new corpus of source code to be tested. The computing device executes the new corpus of source code with one or more test cases from the accessed test set. One or more errors generated by execution of the new corpus of source code with the one or more test cases are accessed. One or more locations in the new corpus of source code associated with the accessed one or more errors are determined. Natural language processing software is executed to generate one or more new tags associated with the one or more locations in the new corpus of source code associated with the accessed one or more errors. An artificial intelligence model associated with the natural language processing model is improved. [Brief description of the drawings]

[0008] [Figure 1] 1 illustrates a networked computing environment 100 according to one embodiment of the present invention.

[0009] [Diagram 2] FIG. 2 is a functional block diagram illustrating a module 200 for tagging a corpus of tests based on fault analysis, according to an embodiment of the present invention.

[0010] [Diagram 3]4 is a flowchart illustrating operational steps that may be performed by hardware components of a hardware appliance according to one embodiment of the present invention.

[0011] [Figure 4A] 4 is a flowchart illustrating operational steps that may be performed by hardware components of a hardware appliance according to an alternative embodiment of the present invention. [Figure 4B] 4 is a flowchart illustrating operational steps that may be performed by hardware components of a hardware appliance according to an alternative embodiment of the present invention.

[0012] [Diagram 5] 4 is a flowchart illustrating operational steps that may be performed by hardware components of a hardware appliance according to an alternative embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Embodiments of the present disclosure relate to one or more methods, systems, and computer program products for tagging a corpus of software tests based on failure analysis. Systematic, continuously revalidated tagging of tests is useful in many tasks that may accompany software development, including source code verification, source code debugging, clustered testing, and the like. Natural language processing software, such as that utilizing artificial intelligence software including large language models, streamlines the task of automatically tagging tests. Embodiments of the present invention rely or rely on previous runs of embodiments that tag tests using regression analysis of results from previous runs, where results from previous runs are refined and further elaborated upon. Regression analysis may also be applied to previous versions, rollouts, etc. (including updates to tests that verify, drive, or exercise source code) of source code or other corpora of source code that may be related in some manner.

[0014] Embodiments of the present disclosure may be utilized in connection with any environment in which source code is developed, tested, accessed, edited, modified, etc., for example, within an interactive development environment that allows multiple developers, project managers, etc. to develop source code (and provides, for example, the ability to view, edit, compile, execute, test, and debug the source code). Embodiments of the present disclosure streamline the process of software development by allowing individuals involved in the development process to systematically tag a corpus of tests. In general, embodiments of the present disclosure may be implemented in different manners, although they are contemplated by embodiments of the present invention. Embodiments of the present disclosure may be implemented in different manners, although they are contemplated by embodiments of the present invention disclosed herein, for example, as a plug-in to an interactive development environment, as a standalone application, as part of a web browser, or in any other manner.

[0015] Various aspects of the disclosure are described by text, flow charts, block diagrams of computer systems, and / or block diagrams of machine logic included in embodiments of computer program products (CPPs). For any flow chart, depending on the technology involved, operations may be performed in a different order than shown in a given flow chart. For example, two operations shown in successive flow chart blocks may be performed in reverse order, as a single integrated step, simultaneously, or in an at least partially overlapping manner in time, again depending on the technology involved.

[0016] A computer program product embodiment ("CPP embodiment" or "CPP") is a term used in this disclosure to describe any set of one or more storage media (also referred to as "media") collectively contained within a set of one or more storage devices that collectively contain machine-readable code corresponding to instructions and / or data for performing computer operations specified in a given CPP claim. A "storage device" is any tangible device that can hold and store instructions for use by a computer processor. The computer-readable storage medium may be, but is not limited to, an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, a mechanical storage medium, or any suitable combination of the foregoing. Some known types of storage devices that include these media include diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices (such as punch cards, or pits / lands formed on the major surface of a disk), or any suitable combination of the foregoing. A computer-readable storage medium, as the term is used in this disclosure, should not be construed as storage in the form of a transient signal per se, such as electric waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide, light pulses passing through a fiber optic cable, electrical signals transmitted through wires, and / or other transmission media. As will be appreciated by those skilled in the art, data is typically moved at some infrequent time during the normal operation of the storage device, such as during access, defragmentation, or garbage collection, but this does not make the storage device transient, since the data is not transient while it is stored.

[0017] The computing environment 100 includes an example of an environment for the execution of at least some of the computer code involved in the execution of the method of the present invention, for example associated with a module 200 for tagging a corpus of tests based on failure analysis. In addition to the module 200, the computing environment 100 includes, for example, a computer 101, a wide area network (WAN) 102, an end user device (EUD) 103, a remote server 104, a public cloud 105, and a private cloud 106. In this embodiment, the computer 101 includes a processor set 110 (including processing circuitry 120 and cache 121), a communication fabric 111, a volatile memory 112, a persistent storage 113 (including an operating system 122 and modules 200 identified above), a peripheral device set 114 (including a user interface (UI) device set 123, storage 124, and an Internet of Things (IoT) sensor set 125), and a network module 115. The remote server 104 includes a remote database 130. The public cloud 105 includes a gateway 140 , a cloud orchestration module 141 , a set of host physical machines 142 , a set of virtual machines 143 and a set of containers 144 .

[0018] Computer 101 may take the form of a desktop computer, a laptop computer, a tablet computer, a smartphone, a smartwatch or other wearable computer, a mainframe computer, a quantum computer, or any other form of computer or mobile device now known or later developed that can execute programs, access a network, or query a database, such as remote database 130. As is well understood in the field of computer technology, and depending on the technology, execution of a computer-implemented method may be distributed among multiple computers and / or among multiple locations. However, in this presentation of computing environment 100, in order to keep the presentation as simple as possible, the detailed discussion focuses on a single computer, specifically computer 101. Although computer 101 is not shown in the cloud in FIG. 1, it may be located in the cloud. However, computer 101 is not required to be in the cloud except to any extent that may be categorically indicated.

[0019] Processor set 110 includes one or more computer processors of any type now known or later developed. Processor set 110 may alternatively be referred to herein as "one or more computing devices," which may refer to one or more CPUs, microchips, integrated circuits, embedded systems, or equivalents now existing or later. Processing circuitry 120 may be distributed across multiple packages, e.g., multiple tailored integrated circuit chips. Processing circuitry 120 may implement multiple processor threads and / or multiple processor cores. Cache 121 is memory located within the processor chip package and is typically used for data or code that should be available for quick access by threads or cores executing on processor set 110. Cache memory is typically organized into multiple levels depending on relative proximity to the processing circuitry. Alternatively, some or all of the cache for a processor set may be located "off-chip." In some computing environments, processor set 110 may be designed to operate with qubits and perform quantum computing.

[0020] Computer-readable program instructions are typically loaded into the computer 101 to cause a sequence of operational steps to be executed by the processor set 110 of the computer 101, resulting in a computer-implemented method in which the resulting instructions instantiate the methods specified in the computer-implemented method flowcharts and / or descriptions contained herein (collectively, the "methods of the present invention"). These computer-readable program instructions are stored in various types of computer-readable storage media, such as the cache 121 and other storage media described below. The program instructions and associated data are accessed by the processor set 110 to control and direct the execution of the methods of the present invention. In the computing environment 100, at least some of the instructions for executing the methods of the present invention may be stored in a module 200 in persistent storage 113.

[0021] Communications fabric 111 is the signal-conducting paths that allow various components of computer 101 to communicate with one another. Typically, this fabric is made up of switches and electrically conductive paths, such as those that make up buses, bridges, physical input / output ports, and the like. Other types of signal communication paths may be used, such as fiber optic and / or wireless communication paths.

[0022] Volatile memory 112 is any type of volatile memory now known or developed in the future. Examples include dynamic type random access memory (RAM) or static type RAM. Typically, volatile memory 112 is characterized by random access, although this is not required unless expressly indicated. In computer 101, volatile memory 112 is located in a single package and is internal to computer 101, although alternatively or additionally, volatile memory may be distributed across multiple packages and / or located external to computer 101.

[0023] The persistent storage 113 is any form of non-volatile storage for a computer, now known or later developed. The non-volatility of this storage means that the stored data is maintained regardless of whether power is provided to the computer 101 and / or directly to the persistent storage 113. The persistent storage 113 may be a read-only memory (ROM), but typically at least a portion of the persistent storage allows data to be written, data to be deleted, and data to be rewritten. Some well-known forms of persistent storage include magnetic disks and solid-state storage devices. The operating system 122 may take several forms, such as various known proprietary operating systems, or an open source Portable Operating System Interface type operating system that uses a kernel. The code contained within the module 200 typically includes at least some of the computer code involved in carrying out the methods of the present invention.

[0024] The peripheral device set 114 includes a set of peripheral devices of the computer 101. Data communication connections between the peripheral devices and other components of the computer 101 may be implemented in various ways, such as Bluetooth connections, Near Field Communication (NFC) connections, connections made by cables (such as Universal Serial Bus (USB) type cables), insertion type connections (e.g., Secure Digital (SD) cards), connections made through local area communication networks, and even connections made through wide area networks such as the Internet. In various embodiments, the UI device set 123 may include components such as display screens, speakers, microphones, wearable devices (such as goggles and smart watches), keyboards, mice, printers, touch pads, game controllers, and haptic devices. The storage 124 is external storage, such as an external hard drive, or insertable storage, such as an SD card. The storage 124 may be persistent and / or volatile. In some embodiments, the storage 124 may take the form of a quantum computing storage device for storing data in the form of qubits. In embodiments where computer 101 is required to have a large amount of storage (e.g., computer 101 stores and manages a large database locally), this storage may be provided by a peripheral storage device designed to store very large amounts of data, such as a storage area network (SAN) shared by multiple geographically distributed computers. IoT sensor set 125 consists of multiple sensors that may be used in Internet of Things applications. For example, one sensor may be a thermometer and another sensor may be a motion detector.

[0025] The network module 115 is a collection of computer software, hardware, and firmware that enables the computer 101 to communicate with other computers over the WAN 102. The network module 115 may include hardware such as a modem or Wi-Fi® signal transceiver, software for packetizing and / or depacketizing data for communication network transmission, and / or web browser software for communicating data over the Internet. In some embodiments, the network control and network forwarding functions of the network module 115 are performed on the same physical hardware device. In other embodiments (e.g., those utilizing software-defined networking (SDN)), the control and forwarding functions of the network module 115 are performed on physically separate devices such that the control function manages several different network hardware devices. Computer-readable program instructions for carrying out the methods of the present invention may be downloaded to the computer 101 from an external computer or external storage device, typically through a network adapter card or network interface included in the network module 115.

[0026] WAN 102 is any wide area network (e.g., the Internet) capable of communicating computer data over non-local distances by any technology for communicating computer data now known or developed in the future. In some embodiments, WAN 102 may be replaced and / or supplemented by a local area network (LAN) designed to communicate data between devices located within a local area, such as a Wi-Fi® network. WANs and / or LANs typically include computer hardware, such as copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and edge servers.

[0027] End-user device (EUD) 103 is any computer system used and controlled by an end user (e.g., a customer of the enterprise operating computer 101) and may take any of the forms described above in connection with computer 101. EUD 103 typically receives useful and useful data from the operation of computer 101. For example, in a hypothetical case in which computer 101 is designed to provide recommendations to the end user, the recommendations would typically be communicated from network module 115 of computer 101 over WAN 102 to EUD 103. In this manner, EUD 103 can display or otherwise present the recommendations to the end user. In some embodiments, EUD 103 may be a client device, such as, for example, a thin client, a heavy client, a mainframe computer, and a desktop computer.

[0028] Remote server 104 is any computer system that provides at least some data and / or functionality to computer 101. Remote server 104 may be controlled and used by the same entity that operates computer 101. Remote server 104 represents a machine that collects and stores useful and useful data for use by other computers, such as computer 101. For example, in the hypothetical case where computer 101 is designed and programmed to provide recommendations based on historical data, this historical data may be provided to computer 101 from remote database 130 of remote server 104.

[0029] A public cloud 105 is any computer system available for use by multiple entities that provides on-demand availability of computer system resources and / or other computer functionality, particularly data storage (cloud storage) and computing power, without direct active management by users. Cloud computing typically leverages the sharing of resources to achieve coherence and economies of scale. The direct and active management of computing resources of the public cloud 105 is performed by computer hardware and / or software of a cloud orchestration module 141. The computing resources provided by the public cloud 105 are typically implemented by virtual computing environments running on various computers that make up the computers of a host physical machine set 142, which is a universe of physical computers within and / or available to the public cloud 105. A virtual computing environment (VCE) typically takes the form of a virtual machine from a virtual machine set 143 and / or a container from a container set 144. It is understood that these VCEs may be stored as images and may be transferred among and between a host of different physical machines, either as images or after instantiation of the VCE. Cloud orchestration module 141 manages the transfer and storage of images, deploys new instantiations of VCEs, and manages active instantiations of VCE deployments. Gateway 140 is a collection of computer software, hardware, and firmware that enables public cloud 105 to communicate over WAN 102.

[0030] Some further description of Virtualized Computing Environments (VCEs) is now provided. A VCE may be stored as an image. A new active instance of a VCE may be instantiated from the image. Two well-known types of VCEs are virtual machines and containers. A container is a VCE that uses operating system level virtualization. This refers to a feature of an operating system where the kernel allows the existence of multiple isolated user space instances called containers. These isolated user space instances typically behave as real computers from the perspective of the programs running within them. A computer program running on a normal operating system can utilize all of the resources of that computer, such as attached devices, files and folders, network shares, CPU power and quantifiable hardware capabilities. However, a program running inside a container can only use the contents of the container and of the devices assigned to the container, a feature known as containerization.

[0031] A private cloud 106 is similar to a public cloud 105, except that the computing resources are only available for use by a single enterprise. Although the private cloud 106 is shown in communication with the WAN 102, in other embodiments, the private cloud may be completely disconnected from the Internet and only accessible through a local / private network. A hybrid cloud is a composite of multiple clouds of different types (e.g., private, community or public cloud types), often each implemented by a different vendor. While each of the multiple clouds remains a separate, discrete entity, the larger hybrid cloud architecture is bound together by standardized or proprietary technologies that enable orchestration, management and / or data / application portability between the multiple constituent clouds. In this embodiment, both the public cloud 105 and the private cloud 106 are part of a larger hybrid cloud.

[0032] FIG. 2 is a functional block diagram illustrating a module 200 for tagging a corpus of tests based on fault analysis, according to an embodiment of the present invention. In overview, FIG. 2 illustrates an interactive development environment 210, a tagging module 260, and a network 299. The interactive development environment 210 allows individuals involved in the source code development process to perform various functions in relation to the source code, including accessing, storing, editing, compiling / interpreting, testing, and debugging different versions of the source code. An embodiment of the present invention may be utilized in relation to various types of source code, including compiled, interpreted, scripted bytecode, or even previously developed software, all of which require testing and may benefit from automatic tagging, as described in an embodiment of the present invention. The various functions provided by the interactive development environment 210 may enable a developer, project manager, or other user interested in the development process to check in, check out, store, edit, compile / interpret, run, and debug source code, as well as perform other functions in relation to different versions of the source code stored by the interactive development environment 210. Interactive development environment 210 actually provides an interface for one or more users to perform various functions in connection with source code and / or software. Interactive development environment 210 may be implemented as any type of computer software (and, in various embodiments, associated computer hardware) that performs the functions described herein. As further described herein, interactive development environment 210 interfaces with tagging module 260 to systematically and continuously tag a corpus of tests.

[0033] The tagging module 260 is also shown in FIG. 2 and is operatively connected to the interactive development environment 210, either directly or via a network 299. Alternatively, the tagging module 260 is integrated with the interactive development environment 210, for example, via a plug-in to the interactive development environment 210, access to / through a common web browser, or other equivalent means. Generally, and as further described in connection with embodiments of the invention disclosed herein, the tagging module 260 is responsible in various embodiments of the invention for automatically tagging a corpus of tests using natural language processing software or other techniques to systematically generate tags for various versions of the tests. As one skilled in the art will appreciate, the "tags" may be used in various ways to flag modifications, different versions, degrees of change, comments, etc. in various different corpora of source code available from the interactive development environment 210. The tags may be used by developers, for example, to assist in manually finding bugs in the source code, to generate cluster tests, etc., or to assist in automatically debugging the source code. In embodiments of the invention, as described further herein, tagging module 260 operates via regression to utilize results of previous runs of tagging module 260 to obtain the best results (i.e., obtain the "best" tags for a corpus of tests) in a fast and efficient manner. Tagging module 260 may be implemented as any type of computer hardware (and associated computer hardware for performing the functions described, in various embodiments). In various embodiments of the invention, tagging module 260 may, for example, access source code available from interactive development environment 210 and access features of interactive development environment 210 to automatically compile, interpret, or otherwise execute the source code and systematically tag tests accessed from interactive development environment 210, as described further herein.

[0034] As further shown in FIG. 2, in various embodiments of the present invention, the tagging module 260 and the interactive development environment 210 are connected to and through a network 299. In various embodiments of the present invention, the network 299 represents any type of computer network, such as, for example, a local area network (LAN), a wide area network (WAN) such as the Internet, including wired, wireless or fiber optic connections. In various embodiments, the network 299 is substantially the same as the WAN 102 described herein in connection with FIG. 1. In general, the network 299 may be any combination of connections and protocols that support communication between the interactive development environment 210 and the tagging module 260 according to embodiments of the present invention. In further embodiments of the present invention, the network 299 may represent an internal bus associated with a single or multi-core processor that executes both the interactive development environment 210 and the tagging module 260 (e.g., in embodiments in which the interactive development environment 210 and the tagging module 260 are integrated).

[0035] 2 in more detail, interactive development environment 210 represents computer software (and, in various embodiments, associated computer hardware) for performing various functions in connection with source code, including accessing, storing, editing, compiling / interpreting, executing, and debugging a corpus of source code. As will be appreciated by those skilled in the art, interactive development environment 210 provides various facilities necessary for one or more developers, project managers, etc. to develop, test, execute, debug, etc. software. As described elsewhere herein, in embodiments of the invention, interactive development environment 210 is operatively connected to or integrated with tagging module 260.

[0036] In various embodiments of the present invention, interactive development environment 210 includes one or more of a source code repository 213 , a source code editor 215 , build automation 218 , and a debugger 223 .

[0037] Source code repository 213 represents software and / or hardware for storing, accessing, checking in, checking out, saving, etc., different corpora of source code that have been made available by developers (e.g., via uploading, via source code editor 215, or otherwise). Source code repository 213 stores the source code in an accessible format. Those skilled in the art will appreciate that as multiple developers, project managers, etc. continue to edit the source code, it is important to store multiple versions of the uploaded corpus of source code in cases where it is necessary to roll back to a different version (e.g., due to important sections being missing, bugs being present, new objects being poorly designed, etc.). If a downstream developer makes a major mistake, it is a simple matter of simply rolling back to a previous version using source code repository 213, rather than rewriting the entire corpus of source code. However, as described in the context of the disclosed embodiments, when a new corpus of source code, version, etc. is checked into the source code repository, it may be necessary to "tag" significant tests associated with the source code, e.g., via concept, execution, and / or logic. The time-consuming task of tagging existing or new corpora of tests of source code in repository 213 may be streamlined in connection with embodiments of the present invention. In embodiments of the present invention, source code repository 213 stores one or more test sets (each including one or more test cases for testing the corpus of source code) associated with each corpus of source code, and one or more updates to the test sets (described further herein).

[0038] Source code editor 215 represents software and / or hardware for providing a facility for users, developers, project managers, etc. to develop and edit source code in interactive development environment 210. Source code editor 215 may also provide various options for developers, project managers, etc. to check in source code, request compilation of source code, run source code, debug source code, etc., as described in connection with source code repository 213, source code editor 215, build automation 218, and debugger 223.

[0039] Build automation 218 represents software and / or hardware for automatically (or upon request) compiling a corpus of source code stored in source code repository 213. Depending on the type of source code under consideration, build automation 218 may be associated with a compiler, an interpreter, or in another equivalent manner, to convert high-level human-readable source code into machine language for execution by a computing device. In the context of embodiments of the present invention, build automation 218 may be invoked by tagging module 260 to request compilation and / or execution of a corpus of source code for further utilization as described in embodiments disclosed herein (e.g., for use with debugging or automatic tagging of a corpus of tests). Build automation 218 in various embodiments of the present invention is associated with one or more compilers / debuggers.

[0040] Debugger 223 represents software and / or hardware for automatically debugging software or to aid in manual debugging by a developer. Debugger 223 may be associated with build automation 218 and / or a compiler / interpreter (not shown). In the context of compiling, interpreting, or otherwise executing a corpus of source code, the debugger may track one or more errors, including, for example, compilation errors, run-time errors, or other errors generated by a compiler / interpreter (not shown, but associated with build automation 218). Errors tracked by debugger 223 may be transmitted to tagging module 260 for further use. The tracked errors may be used in various manners in the context of embodiments of the present invention (described more fully herein).

[0041] Continuing with the elements of FIG. 2, tagging module 260 represents computer software (and, in various embodiments, associated computer hardware) for automatically tagging a corpus of tests. As described elsewhere herein, a corpus of source code, including different versions, updates, edits, etc. of an associated source code project, is maintained by interactive development environment 210. Those skilled in the art will appreciate that interactive development environment 210 stores different versions of source code, etc., because, for example, if a bug in the current version of the source code prevents the current version from providing accurate results, etc., it may be necessary to roll back to a previous version. Generally, tagging module 260 presents techniques for systematically tagging a corpus of tests associated with different versions of source code, and refines and generates new tags based on previous tag creations via regression execution, as further described herein. The new tags generated by tagging module 260 may be displayed to a user via a user interface associated with interactive development environment 210, such as source code editor 215.

[0042] In various embodiments of the present invention, tagging module 260 includes one or more of tag database 262, test case module 265, source code access and execution module 268, error module 271, natural language tagging module 276, tag database update module 278, test execution history 279, user access module 280, and machine learning improvement module 285.

[0043] Tag database 262 represents software and / or hardware for storing, accessing, and making available to interactive development environment 210 one or more "tags" associated with different versions of a test. In various embodiments, the tags stored by tag database 262 are associated with one or more test cases known to have executed without error against the original corpus of source code available from interactive development environment 210, or may be generated by other iterations of embodiments of the invention. In embodiments of the invention, execution begins by a computing device accessing from tag database 262 one or more tags associated with one or more test cases known to have executed without error against the original corpus of source code. Beginning execution with tags associated with test cases known to have executed without error against the original corpus of source code provides efficient and effective tagging of the relevant corpus of tests, and the associated "tags" are used as starting points for execution of other aspects or iterations of the invention. In embodiments of the invention, tag database 262 is also responsible for storing one or more newly generated tags that are further utilized in regression execution of embodiments of the invention to generate newer or more detailed tags for other corpora of tests. As described further herein, as new tags are generated by iterations of embodiments of the present invention, they are stored by tag database 262 for further use.

[0044] Test case module 265 represents software and / or hardware for accessing, downloading, modifying, updating, uploading, etc., one or more test sets from interactive development environment 210. As described in connection with tag database 262 and other embodiments disclosed herein, a test set is associated with one or more accessed tags. Each test set accessed from interactive development environment 210 may include one or more test cases for testing a corpus of source code (e.g., including testing variables, methods, interfaces, inputs, outputs, etc., and in embodiments of the present invention, testing concepts, relationships, properties, resources that may be verified / driven / executed by the tests). In one embodiment of the present invention, when execution begins, a test set accessed from test case module 265 is known to include one or more test cases that are known to execute without error against the original corpus of source code accessed from interactive development environment 210. In various embodiments of the present invention, test case module 265 also provides functionality for accessing a new test set of one or more new test cases associated with one or more generated new tags (described further herein). A "new test set" as described herein refers to an update, change, or modification from a "test case known to run without error." The new test set associated with the generated new tag or tags provides a more extensive test of the new corpus of source code (i.e., the new test set, comprised of one or more test cases, may provide more detailed information about the type of error encountered in debugging because the new test set has previously been associated with other corpora / bodies of source code in which similar errors have occurred in the past. The other corpora / bodies may be previous versions of the new corpus of source code, completely unrelated source code, compiled software, or some combination of these).New test sets may be derived automatically (e.g., by machine learning models) from other corpora / bodies of source code created manually, etc. Execution of new embodiments of the invention with newer versions of source code and associated tests is a good starting point, since errors encountered by previous versions of source code and associated tests are likely to be similar to new errors encountered by newer versions of source code and associated tests. As execution of embodiments of the invention proceeds, test case module 265 may also be associated with the generation of newer test sets, which are associated with one or more new errors available from error module 271, etc. In connection with these embodiments, new locations in the new corpus of source code are associated with one or more new errors accessed, and one or more newer tags. As will be appreciated by those skilled in the art, continued execution of embodiments of the invention will provide finer and more detailed results through regression execution. In various embodiments of the invention, one or more newly generated new tags may be further associated with test execution history 279 to provide developers, project managers, etc. with further information regarding execution, errors, tags, etc. of different versions of source code.

[0045] The source code access and execution module 268 represents software and / or hardware for accessing the corpus of source code from the interactive development environment 210. The source code stored by the interactive development environment 210 may be in different formats, such as uncompiled, compiled or partially compiled, and may be of different types, such as interpreted, scripted, etc. As the developer continues to edit, modify and upload a new corpus of source code to the interactive development environment 210, the interactive development environment continues to store (also referred to as "checking in") previous versions of the source code, since a "rollback" to a previous version may be necessary, or for other reasons. Different versions of the source code are utilized as described herein. For clarity, the "original corpus of source code" refers to the first version of such source code under consideration, while the "new corpus of source code" refers to another corpus of source code, including edits, improvements, modifications, etc., to the "original corpus of source code." In an embodiment of the invention, newer and still newer edits, improvements, modifications, etc. to each successive version may be stored, accessed, tagged, etc. by interactive development environment 210. Through regression execution of an embodiment of the invention, the newer and still newer versions of the source code may be accessed and modified in connection with further iterations of the invention. In an alternative embodiment of the invention, the new corpus of source code is simply completely new code (relative to the original corpus of source code). In connection with the newer and still newer versions of source code, execution of an embodiment of the invention serves to update both the corpus of tests to reflect the newer and still newer versions of source code, etc.

[0046] The source code access and execution module 268 also functions to request execution of various corpora of source code, versions, etc. by the interactive development environment 210. The execution requested by the source code access and execution module 268 relates to the execution of a particular corpus of source code with a particular set of test cases from a particular test set. As described herein, the various sets of test cases may be referred to as "one or more test cases known to execute without error," "a new test set," "a newer test set," etc. As embodiments of the present invention are executed, the described test cases are executed via a regression execution operation as described further herein due to new tags related to the update of the corpus of tests. As the execution proceeds, execution may be requested with new test cases from the new test set or a newer test set, etc. To tag the corpus of tests in an efficient manner, one skilled in the art will appreciate that each successive test set is an update or newly generated iteration of the previous test set.

[0047] The error module 271 represents software and / or hardware for accessing one or more errors associated with the execution of various corpora of source code with different test sets. The interactive development environment 210, after being requested by the source code access and execution module 268 to execute a particular corpus of source code with a particular test set, accesses one or more errors generated by the execution of the particular corpus of source code. These errors may include, as non-limiting examples, compilation errors, run-time errors, logic errors, incorrect results, etc. As one or more errors are generated by the execution of the various corpora of source code, these errors are accessed by the error module 271 for further use as described herein. Also, as described herein, the error module 271 is also associated with detecting new errors detected by the generation of the execution of the new corpus of source code with one or more new test cases, detecting further errors associated with the execution of the various corpora of source code, etc. In general, errors may be associated with one or more newly edited sections of a different version of the source code relative to a previous version (e.g., because a developer mistypes a name of a method or variable in the new section of the source code, etc.).

[0048] Error module 271, in conjunction with source code access and execution module 268 and other modules described in FIG. 2, is for determining one or more locations within various corpora of source code associated with an accessed error. Continuing with the previous example, if a developer mistypes a variable "foo" resulting in a runtime error, error module 271 identifies where the mistyped variable "foo" is located within the corpus of source code. As execution proceeds, error module 271 may similarly determine one or more new locations within a new corpus of source code associated with a new error, and so on. In various embodiments of the invention, error module 271 may simply receive from interactive development environment 210 the location of the error within the corpus of source code, as well as which section of the source code received the error and what the variable / method or equivalent means is.

[0049] Natural language tagging module 276 represents software and / or hardware for the generation of new tags for various corpora of tests. After errors are accessed (or otherwise detected) by error module 271 and one or more locations associated with the errors are determined, natural language tagging module 276 automatically tags these potential error regions in the corpus of source code, and in embodiments of the present invention, also tags one or more tests in the corpus of tests associated with the errors. In various embodiments of the present invention, natural language tagging module 276 may rely, at least in part, on various development metadata in tagging the corpus of tests, e.g., development metadata associated with the source code development process, preferred edits by different developers, etc.

[0050] In various embodiments of the invention, the natural language tagging module 276 may rely on artificial intelligence-based models (such as large-scale language models) to easily and quickly generate new tags for the associated corpus of tests. The large-scale language models may be trained, retrained, etc. with a previously tagged corpus of source code and sets of tests so that the rapid generation of new tags occurs in an easy manner. The large-scale language models used may be based on general use with source code, but may be improved by training, retraining, etc. with a similar specifically tagged corpus of tests (as described further in connection with the machine learning improvement module 285). In various embodiments of the invention, as the regression run continues, the natural language tagging module 276 continues to generate ever newer tags for further downstream versions of the tests, which ever newer tags provide more detailed fault analysis, etc. of the new corpus of source code available from the interactive development environment 210 (as described further herein). As the "new tags", "newer tags", "even newer tags", etc. are created, they are committed to the tag database 262 for further use with future iterations of embodiments of the invention.

[0051] The tag database update module 278 represents software and / or hardware for updating the records of tags generated in connection with the iterative regression execution of the present invention. As execution continues, the records of tags are utilized to create more detailed tags (as described herein).

[0052] Test execution history 279 represents software and / or hardware for storing records of execution of previous iterations of the disclosed embodiments. In various embodiments, test execution history 279 includes records of distinct source code, associated corpora of tests (i.e., test cases), associated errors, and other data described herein for use in embodiments of the invention.

[0053] User access module 280 represents software and / or hardware for users to access and view one or more new tags, a corpus of tests, one or more determined locations within the new corpus of source code associated with the accessed one or more errors, and other data as described in connection with embodiments of the present invention. Various users (developers, project managers, etc.) may access the one or more errors through interactive development environment 210 (which is in communication with user access module 280) or in another manner.

[0054] Machine learning improvement module 285 represents software and / or hardware for improving the operation of a machine learning module utilized in connection with embodiments of the present invention (e.g., in connection with natural language tagging module 276). In embodiments in which natural language tagging module 276 utilizes a large language model, for example because the large language model may have previously been trained primarily on a corpus of fiction and little source code, machine learning improvement module 285 may utilize a corpus of source code, test cases, etc., specifically to "improve" the large language model for better utilization in connection with embodiments of the present invention. For example, a tagged corpus of test cases may be used, for example, to train or retrain the large language model. In embodiments of the present invention in which different types of artificial intelligence, neural networks, etc. are utilized in connection with natural language processing software (described in connection with natural language tagging module 276), machine learning improvement module 285 may be used to improve the functionality of those things.

[0055] FIG. 3 is a flow chart illustrating operational steps that a hardware component, a plurality of hardware components, and / or a hardware appliance may perform according to an embodiment of the present invention. As shown in FIG. 3, in step 310, tag database 262 accesses one or more tags associated with one or more test cases known to execute without errors on an original corpus of source code. In step 320, test case module 265 accesses a test set associated with one or more accessed tags, the test set including one or more test cases. In step 330, source code access and execution module 218 accesses a new corpus of source code to be tested. In step 340, source code access and execution module 268 executes the new corpus of source code with one or more test cases from the accessed test set. In step 350, error module 271 accesses one or more errors generated by the execution of the new corpus of source code with one or more test cases. In step 360, error module 271 determines one or more locations in the new corpus of source code associated with the accessed one or more errors. In step 370, the natural language tagging module 276 executes natural language processing software to generate one or more new tags associated with one or more locations in the new corpus of source code associated with the accessed error or errors. Execution of one embodiment as shown in Figure 3 presents the benefits of systematically "tagging" a new corpus of tests, and presenting time savings, operational efficiency and accuracy to developers, project managers, and the like, and any others interested in source code or other software, with respect to these automatically generated tags.

[0056] 4A and 4B are flow charts illustrating alternative operational steps that a hardware component, multiple hardware components, and / or hardware appliance may perform according to an embodiment of the present invention. As shown in FIG. 4A, in step 410, tag database 262 accesses one or more tags associated with one or more test cases known to execute without errors on an original corpus of source code. In step 420, test case module 265 accesses a test set associated with one or more accessed tags, the test set including one or more test cases. In step 430, source code access and execution module 218 accesses a new corpus of source code to be tested. In step 440, source code access and execution module 268 executes the new corpus of source code with one or more test cases from the accessed test set. In step 450, error module 271 accesses one or more errors generated by the execution of the new corpus of source code with one or more test cases. In step 460, error module 271 determines one or more locations in the new corpus of source code associated with the accessed one or more errors. In stage 470, the natural language tagging module 276 executes natural language processing software to generate one or more new tags associated with one or more locations in the new corpus of source code associated with the accessed one or more errors.

[0057] As execution continues to Figure 4B, in step 480, tag database 262 accesses the generated one or more new tags associated with one or more locations in the new corpus of source code. In step 490, test case module 265 accesses a new test set associated with the generated new tags, which provides new test cases that provide more extensive testing of the new corpus of source code related to similar categories of errors. Execution of an embodiment as shown in Figures 4A and 4B offers further time savings, operational efficiency and accuracy to developers, project managers, etc., and any others interested in source code or other software, particularly when new versions of source code or software are rolled out, with respect to these automatically generated tags, by also offering the advantage of automatically and more completely tagging a corpus of tests that may be more relevant to the type of automatically generated tag.

[0058] FIG. 5 is a flow chart illustrating alternative operational steps that a hardware component, a plurality of hardware components, and / or a hardware appliance may perform according to an embodiment of the present invention. As shown in FIG. 5, in step 510, tag database 262 accesses one or more tags associated with one or more test cases known to execute without errors on an original corpus of source code. In step 520, test case module 265 accesses a test set associated with one or more accessed tags, the test set including one or more test cases. In step 530, source code access and execution module 218 accesses a new corpus of source code to be tested. In step 540, source code access and execution module 268 executes the new corpus of source code with one or more test cases from the accessed test set. In step 550, error module 271 accesses one or more errors generated by the execution of the new corpus of source code with one or more test cases. In step 560, error module 271 determines one or more locations in the new corpus of source code associated with the accessed one or more errors. At stage 570, the natural language tagging module 276 executes natural language processing software to generate one or more new tags associated with one or more locations in the new corpus of source code associated with the accessed one or more errors. At stage 580, the machine learning improvement module 285 improves the artificial intelligence model associated with the natural language processing module. As described herein, execution of one embodiment as shown in FIG. 5 presents the advantages of generating new tags for a new corpus of source code (saving time, improving operational efficiency and improving accuracy) and automatically improving the artificial intelligence model used in multiple techniques. The improvement of the artificial intelligence model is useful in conjunction with the execution of other embodiments of the invention, for example, improving the accuracy and speed of predictions.

[0059] Based on the foregoing, a method, system and computer program product have been disclosed. However, numerous modifications and substitutions can be made without departing from the scope of the invention. Thus, the present invention has been disclosed by way of example and not limitation.

Claims

1. 1. A method for tagging a corpus of tests based on failure analysis using a computing device, comprising: accessing, by the computing device, one or more tags associated with one or more test cases known to run without error against the original corpus of source code; accessing, by the computing device, a test set associated with the one or more accessed tags, the test set including one or more test cases; accessing, by said computing device, a new corpus of source code to be tested; executing, by the computing device, the new corpus of source code using one or more test cases from the accessed test set; accessing one or more errors generated by execution of the new corpus of source code with the one or more test cases; determining one or more locations within the new corpus of source code associated with the accessed one or more errors; and executing, by the computing device, natural language processing software to generate one or more new tags associated with the one or more locations in the new corpus of source code associated with the accessed one or more errors. A method comprising:

2. accessing, by the computing device, the generated one or more new tags associated with the one or more locations within the new corpus of source code; and accessing, by the computing device, a new test set associated with the generated one or more new tags, the new test set providing one or more new test cases that provide more extensive testing of the new corpus of source code associated with similar categories of errors. The method of claim 1 further comprising:

3. running, by the computing device, the new corpus of source code with one or more new test cases from the new test set; accessing one or more new errors generated by execution of the new corpus of source code with one or more new test cases; determining one or more new locations within the new corpus of source code associated with the accessed one or more new errors; and executing, by the computing device, natural language processing software to generate one or more further new tags associated with the one or more new locations in the new corpus of source code associated with the accessed one or more new errors, the one or more further new tags providing a more detailed fault analysis of the new corpus of source code. The method of claim 2 , further comprising:

4. The method of claim 2 , wherein the new test set is derived from another corpus of source code that has previously encountered similar errors.

5. generating a still newer test set, the still newer test set being associated with the one or more new errors, the new locations in the new corpus of source code associated with the accessed one or more new errors, and the one or more still new tags; The method of claim 3 further comprising:

6. The method of claim 1 , wherein the new corpus of source code to be tested is an update or new version of the original corpus of source code.

7. 7. The method of claim 6, wherein the one or more errors generated by execution of the new corpus of source code are associated with one or more edits in the new corpus of source code relative to the original corpus of source code.

8. The method of claim 1 , wherein the original corpus of source code is compiled or interpreted.

9. 2. The method of claim 1 , wherein the one or more new tags and the determined one or more locations within the new corpus of source code associated with the accessed one or more errors are displayed to one or more users via an interface.

10. 2. The method of claim 1 , wherein the new corpus of source code, the one or more new tags, and the determined one or more locations within the new corpus of source code are used to improve an artificial intelligence model associated with the natural language processing software.

11. The method of claim 1 , wherein the new corpus of source code is submitted to a repository.

12. The method of claim 11 , wherein the repository includes source code, one or more test sets, and one or more updates to the test sets.

13. 2. The method of claim 1, wherein the one or more new tags associated with the one or more locations in the new corpus of source code associated with the accessed one or more errors are generated based on a test execution history.

14. The method of claim 1 , wherein the natural language processing software is associated with a large scale language model.

15. 15. The method of claim 1, wherein the natural language processing software relies, at least in part, on development metadata when generating one or more new tags associated with the one or more positions in the new corpus of source code.

16. 1. A method for tagging a corpus of tests based on failure analysis using a computing device, comprising: accessing, by the computing device, one or more tags associated with one or more test cases known to run without error against the original corpus of source code; accessing, by the computing device, a test set associated with the one or more accessed tags, the test set including one or more test cases; accessing, by said computing device, a new corpus of source code to be tested; executing, by the computing device, the new corpus of source code using one or more test cases from the accessed test set; accessing one or more errors generated by execution of the new corpus of source code with the one or more test cases; determining one or more locations within the new corpus of source code associated with the accessed one or more errors; executing, by the computing device, natural language processing software to generate one or more new tags associated with the one or more locations in the new corpus of source code associated with the accessed one or more errors; accessing, by the computing device, the generated one or more new tags associated with the one or more locations within the new corpus of source code; and accessing, by the computing device, a new test set associated with the generated one or more new tags, the new test set providing one or more new test cases that provide more extensive testing of the new corpus of source code associated with similar categories of errors. A method comprising:

17. running, by the computing device, the new corpus of source code with one or more new test cases from the new test set; accessing one or more new errors generated by execution of the new corpus of source code with one or more new test cases; determining one or more new locations within the new corpus of source code associated with the accessed one or more new errors; and executing, by the computing device, natural language processing software to generate one or more further new tags associated with the one or more new locations in the new corpus of source code associated with the accessed one or more new errors, the one or more further new tags providing a more detailed fault analysis of the new corpus of source code.

20. The method of claim 16, further comprising:

18. 18. The method of claim 16 or 17, wherein the new test set is derived from another corpus of source code that has previously encountered similar errors.

19. 1. A method for tagging a corpus of tests based on failure analysis using a computing device, comprising: accessing, by the computing device, one or more tags associated with one or more test cases known to run without error against the original corpus of source code; accessing, by the computing device, a test set associated with the one or more accessed tags, the test set including one or more test cases; accessing, by said computing device, a new corpus of source code to be tested; executing, by the computing device, the new corpus of source code using one or more test cases from the accessed test set; accessing one or more errors generated by execution of the new corpus of source code with the one or more test cases; determining one or more locations within the new corpus of source code associated with the accessed one or more errors; executing, by the computing device, natural language processing software to generate one or more new tags associated with the one or more locations in the new corpus of source code associated with the accessed one or more errors; and improving, by said computing device, an artificial intelligence model associated with the natural language processing model. A method comprising:

20. 1. A computer system for tagging a corpus of tests based on impairment analysis, comprising: one or more computer processors; One or more computer-readable storage media; and program instructions stored on the one or more computer-readable storage media for execution by at least one of the one or more processors; Equipped with The program instructions include: program instructions for accessing one or more tags associated with one or more test cases known to run without error on the original corpus of source code; program instructions for accessing a test set associated with the one or more accessed tags, the test set including one or more test cases; program instructions for accessing a new corpus of source code to be tested; program instructions for executing the new corpus of source code with one or more test cases from the accessed test set; program instructions for accessing one or more errors generated by executing the new corpus of source code with the one or more test cases; program instructions for determining one or more locations within the new corpus of source code associated with the accessed one or more errors; and program instructions for executing natural language processing software to generate one or more new tags associated with the one or more locations in the new corpus of source code associated with the accessed one or more errors having Computer system.

21. program instructions for accessing the generated one or more new tags associated with the one or more locations within the new corpus of source code; and and program instructions for accessing a new test set associated with the generated one or more new tags, the new test set providing one or more new test cases that provide more extensive testing of the new corpus of source code associated with similar categories of errors.

21. The computer system of claim 20, further comprising:

22. 22. The computer system of claim 20 or 21, wherein the new test set is derived from another corpus of source code that has previously encountered similar errors.

23. 1. A computer program for tagging a corpus of tests based on impairment analysis, comprising: One or more non-transitory computer-readable storage media and program instructions stored on the one or more non-transitory computer-readable storage media capable of carrying out a method. Equipped with The method comprises: accessing, by the computing device, one or more tags associated with one or more test cases known to have executed without error against the original corpus of source code; accessing, by the computing device, a test set associated with the one or more accessed tags, the test set including one or more test cases; accessing, by said computing device, a new corpus of source code to be tested; executing, by the computing device, the new corpus of source code using one or more test cases from the accessed test set; accessing one or more errors generated by executing the new corpus of source code with the one or more test cases; determining one or more locations within the new corpus of source code associated with the accessed one or more errors; Executing, by the computing device, natural language processing software to generate one or more new tags associated with the one or more locations in the new corpus of source code associated with the accessed one or more errors; and improving, by said computing device, an artificial intelligence model associated with the natural language processing model. Including, Computer program.

24. 24. The computer program product of claim 23, wherein the natural language processing software is associated with a machine learning model.

25. 25. The computer program product of claim 24, wherein the machine learning model is a large-scale language model.

Images