Software testing assurance through inconsistent handling detection

Abstract

Tools and methods are described for detecting possible vulnerabilities in automated testing of software under development. Complete line coverage of tests does not necessarily indicate actual coverage of execution scenarios, for example, condition coverage, decision coverage, and other kinds of execution scenario coverage may be lacking even when all source code statements are nominally covered. When source code changes are committed and the corresponding test set remains unchanged, the user is notified that adequate testing is not guaranteed by the current test set. Test assurance code in a development toolchain may mark a pull request, test set, or source code submission to indicate a lack of test assurance. In some cases, for example, a tool that ensures enhancements may require that new or different tests be provided with updated source code as a prerequisite for the source code to be accepted for inclusion in a repository or build.

Description

Background Art

[0001] By feeding input values ​​to a computer program and comparing the behavior and output produced with what is expected, the computer program can be tested. However, except for the simplest programs, the testing of any other program may be incomplete because many aspects of the possible behavior of the program are not exercised during the test. For example, many programs may be given a very large number of different input values. The program may accept email addresses, phone numbers or other inputs from a set of possible inputs, which include both arbitrarily large numbers of valid values ​​and arbitrarily large numbers of invalid values. Moreover, the number of different paths that a program may take during execution can be huge. If a given program contains twenty decision points with corresponding decision conditions, at these decision points, if the condition is true, then execution follows a branch, and when the condition is false, then execution follows another branch, then there are more than one million different paths through the program. Widely available software applications can have thousands of such decision points, not just twenty, and the number of paths through the application code also increases accordingly.

[0002] Computer programs can be tested manually by personnel who feed input values ​​to the program, but many programs are tested at least in part by automated testing now. Test automation tools feed input to the tested program, and automatically check the behavior and output produced for the standard specified in the test suite. Automated testing makes it possible to test the program more thoroughly and in a more consistent manner. But, even with automated testing, some aspects of the program are not exercised or otherwise not fully tested in a certain way and may also occur. For example, even automated testing cannot check whether the program performs correctly for each input by inputting each possible email address, phone number or other input to the program. And, for most (if not all) commercial programs, the automation of the test of the program will not inherently provide the computing resources required for executing each single path, and each single path can be adopted by the program, so many paths are not tested. Summary of the invention

[0003] Some of the teachings herein are motivated by the initial technical challenge of ensuring code quality in conjunction with repository pull requests. An emerging technical challenge is how to improve automated test coverage. In this context, the inventors determine that existing approaches may incorrectly assume that code coverage equals test coverage, and identify the technical challenge of how to usefully utilize test suites to coordinate source code changes. Other technical challenges solved by the innovations taught herein will be apparent to the skilled person through the discussion provided below.

[0004] Some embodiments provide software test assurance (assurance) technology, that is, the technology of enhancing or providing the nature or degree of assurance about software coverage by automated testing. Some of these software test assurance embodiments include a processor, a memory operable to communicate with the processor, and an assurance code. After being executed by the processor, the assurance code receives the code update submission represented as "S-current" (S-current) here. The submission "S-current" includes the part represented as "P-current" (P-current) here, and "P-current" includes the update to the source code. The assurance code determines that the part "P-current" is at least partially covered by the test set (represented as "TS-current" (TS-current)) of the test suite. The assurance code finds out (ascertain) whether the test set "TS-current" is different from the test set "TS-prior", and the test set "TS-prior" at least partially covers the part "P-prior" (P-prior) of the code update submission "S-prior" (S-prior). In this example, "S-Previous" is older than "S-Current", and "P-Current" updates "P-Previous". When the ascertaining ascertains that "TS-Current" is not different from "TS-Previous", the assurance code marks at least one item based on the "TS-Current" test set to indicate a lack of test assurance for the "P-Current" update. That is, the assurance code raises a flag of at least one of the following: "S-Current", "P-Current", or "TS-Current".

[0005] Some software tests ensure that the embodiments provide or use specific actions. For example, the embodiment may receive a code update submission "S-current", the code update submission "S-current" includes a portion "P-current" with one or more lines of source code, and "P-current" includes an update to the source code. The embodiment can determine that the portion "P-current" of the source code is at least partially covered by the test set "TS-current" of the test suite, and can find out whether the test set "TS-current" is different from the test set "TS-previous", and the test set "TS-previous" at least partially covers the portion "P-previous" of the code update submission "S-previous". In this context, "S-previous" is older than "S-current", and "P-current" updates "P-previous". When TS-current is not different from TS-previous, the embodiment may mark at least one item based on the TS-current test set to indicate a lack of test assurance for updates contained in P-current, and may otherwise avoid marking any item on the basis that TS-current is not different from TS-previous relative to P-current. The marked item(s) may be update commits, or committed code, or test sets, or a combination of these items, each marked item being in a past version, a current version, or both.

[0006] Other technical activities related to the teachings of this article will also become apparent to those skilled in the art. The examples given are illustrative only. This summary is neither intended to identify the key features or essential features of the claimed subject matter nor to limit the scope of the claimed subject matter. On the contrary, this summary is provided to introduce some technical concepts in a simplified form, which will be further described in the following specific embodiments. The present invention is defined by the claims, and in the event that this summary conflicts with the claims, the claims shall prevail. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] A more detailed description will be given with reference to the accompanying drawings, which illustrate only selected aspects and are therefore not necessarily exhaustive in terms of coverage or scope.

[0008] Figure 1 is a block diagram illustrating a computer system and also illustrating a configured storage medium;

[0009] Figure 2 is a block diagram illustrating aspects of a computing system enhanced for test assurance, showing test assurance code and other items;

[0010] Figure 3 is a block diagram illustrating examples of different kinds of code coverage for automated testing;

[0011] Figure 4is a block diagram illustrating some scenarios regarding different treatments of source code and testing for a bug fix or feature change;

[0012] Figure 5 is an idealized source code schematic diagram illustrating one or more source code updates that implement a program bug fix, a feature change, or both;

[0013] Figure 6 is an idealized source code diagram illustrating examples of source code coverage by tests in various scenarios;

[0014] Figure 7 is a flow chart illustrating steps in some software test assurance methods;

[0015] Figure 8 is a flow chart further illustrating the steps in some software test assurance methods; and

[0016] Fig. 9 is a block diagram illustrating a repository. DETAILED DESCRIPTION

[0017] Overview

[0018] Innovations can transcend their origins, but understanding the origins of an innovation can help one more fully understand the innovation even when it has evolved beyond its original focus. In the present case, software testing innovations arise in the context of inventors seeking ways to improve software component functionality by improving automated testing of such functionality.

[0019] Many software development teams use code coverage information to detect deficiencies in test coverage. However, there are instances in which lines of code are nominally covered by tests, but the tests do not necessarily cover all execution scenarios. Therefore, when such code is changed, program errors are introduced. Relying purely on superficial code coverage to highlight the lack of test coverage is not enough to properly manage test technical debt in such cases. Some embodiments described herein check for this situation and warn users who can be, for example, developers or testers. When code is added to a project without also adding thorough automated tests for the code, "technical debt" increases.

[0020] When a developer changes a line of code to fix a bug, or add some functionality, the corresponding change in the automated tests is cautious. If no tests covering the edited line are changed, and no new tests covering the line are written, there is a test hole that will be filled, even if the code coverage report shows that the line is covered. Intelligent detection of code updates that lack test coverage recognizes that when a given line of code in an update is subjected to any automated tests, it may be mistakenly considered covered, rather than recognizing the possibility that coverage is only partial and taking action on it.

[0021] The degree of test coverage for code can be defined with respect to the complete space of possible states that may occur during the execution of the code in question. Tests that do not check the entire space of possible states for correctness only provide partial coverage. The reality is that most tests will be partial. But this does not make the tests useless. In fact, in view of the teachings of the present disclosure, this is a reason to focus the testing effort on using limited testing resources in a way that is more likely to reveal shortcomings, and in particular on marking potential test deficiencies as taught herein.

[0022] When code is updated but corresponding test is not changed, there are several possibilities.One possibility is that unchanged test fully covers updated code, for example, covers approximately the same percentage of possible execution states covered in previous version of code.Another possibility is that update is not fully tested.Perhaps there is no test coverage for this update, for example, code line or variable state is added but not tested.Perhaps there is some test coverage for this update, but the reason for this update is that test fails to detect program errors that are now (perhaps) fixed by update, so test also should be updated.In other words, in addition to the widely recognized possibility of program errors of the program under test, it may also happen that program test itself has program errors.

[0023] In the context of submitting code updates to a repository, the submission can take the form of a "pull request," which is a request that updates be pulled into the repository. If an analysis of code updates, test updates (if any), and test coverage as taught herein reveals that the tests have not been carefully updated when the code was updated, the results of the analysis can be presented to the user submitting the pull request. A submission user interface for the pull request can warn or notify the user of inconsistencies between the code updates and the test updates. The user interface can suggest or even require that: (a) one or more tests be modified to cover the scenario that motivated the code change; or (b) one or more new tests be written and activated in the test suite to cover the scenario that motivated the code change.

[0024] Embodiments taught herein help provide software testing assurance. Lack of testing assurance may occur when source code is updated but the corresponding automated tests for the code are not also updated, or when there are technical questions about whether the tests have been adequately updated. When the code is updated, the update may have completed fixing errors that were not caught by the tests. Therefore, updates to the tests can help detect the same or similar errors.

[0025] Some of the embodiments described herein may be viewed by some in a broader context. For example, concepts such as assurance, consistency, coverage, testing, and updating may be considered relevant to specific embodiments. However, from the availability of a broad context, it is not followed to seek exclusive rights to abstract concepts in this article; abstract concepts are not the goal of this disclosure. Instead, this disclosure focuses on providing appropriate specific embodiments whose technical effects fully or partially solve specific technical problems, such as how to prioritize alarms. Other configured storage media, systems, and methods involving assurance, consistency, coverage, testing, or updating are outside the scope of this disclosure. Therefore, with a proper understanding of this disclosure, ambiguity, mere abstraction, lack of technical features, and the accompanying proof problems are also avoided.

[0026] Technical features

[0027] The technical features of the embodiments described herein will be apparent to those of ordinary skill in the art, and will also be apparent to various dedicated readers in a variety of ways. The embodiments address technical activities that are rooted in computing technology, such as automated testing and timely detection and correction of program errors, which affect a wide range of software components. Some software testing assurance embodiments improve the operation of computing systems by automatically detecting inconsistencies in the handling of source code updates relative to code test suite updates, and notifying users that additional or updated tests would be prudent. Detecting potential gaps in test coverage when the tested code is updated and consistently updating the test set helps make systems that use the updated code operate more safely, efficiently, and effectively.

[0028] When the test is missing and not detected, time and resources may be wasted by trying to identify, fix or bypass program errors or security weaknesses, which were originally solved earlier and more effectively when the code with program errors was submitted. Test guarantees can be added to the software development workflow, or improved within such a workflow. For example, by utilizing the guarantee code taught herein to enhance the independent utility (utility) in the development tool, development tool extension or development tool chain, the guarantee of test coverage can be improved. In particular, continuous integration continuous deployment (CICD) workflow will benefit from improved test guarantees. When the code containing program errors is fresh in the developer's mind, for example, when the developer submits the modified code for inclusion in the repository or build in the CICD workflow, it is easier and faster to fix these program errors.

[0029] Other aspects and advantages of the technical characteristics of the present teachings will also be apparent to those skilled in the art from the description provided.

[0030] Acronyms, Abbreviations and Names

[0031] Some acronyms, abbreviations and names are defined below. Others are defined elsewhere herein or do not require definition here to be understood by those skilled in the art.

[0032] ALU: Arithmetic and Logic Unit

[0033] API: Application Programming Interface

[0034] BIOS: Basic Input / Output System

[0035] CD: Compact Disc

[0036] CPU: Central Processing Unit

[0037] DVD: Digital Versatile Disk or Digital Video Disc

[0038] FPGA: Field Programmable Gate Array

[0039] FPU: Floating Point Processing Unit

[0040] GPU: Graphics Processing Unit

[0041] GUI: Graphical User Interface

[0042] DS: Intrusion Detection System, which can usually be, for example, HIDS (Host-based IDS) or NIDS (Network-based IDS)

[0043] IoT: Internet of Things

[0044] IP: Internet Protocol

[0045] LAN: Local Area Network

[0046] OS: Operating System

[0047] RAM: Random Access Memory

[0048] ROM: Read Only Memory

[0049] UEFI: Unified Extensible Firmware Interface

[0050] VM: Virtual Machine

[0051] VS:Visual Programs (a brand of Microsoft Corp.)

[0052] VS Code: Visual Code Program (a brand of Microsoft Corp.)

[0053] WAN: Wide Area Network

[0054] Additional terms

[0055] Reference is made herein to exemplary embodiments such as those shown in the accompanying drawings, and specific language is used herein to describe them. However, changes and further modifications to the features shown herein, as well as additional technical applications of the abstract principles illustrated by the specific embodiments herein, which would occur to a person skilled in the relevant art and possessing the present disclosure, should be considered within the scope of the claims.

[0056] The meaning of the terms is clarified in this disclosure, so these clarifications should be carefully noted when reading the claims. Specific examples are given, but those skilled in the relevant art will understand that other examples may also fall within the meaning of the terms used and fall within the scope of one or more claims. The terms here do not necessarily have the same meaning as they are in general use (especially in non-technical use), in the use of a specific industry, in a specific dictionary or dictionary collection. Figure numerals can be used with various wordings to help illustrate the breadth of the terms. Omitting figure numerals from a given text fragment does not necessarily mean that the text does not discuss the content of the figure. The inventor claims and exercises the right to compile specific and selected dictionaries. Quoted terms are explicitly defined, but it is also possible to define terms implicitly without using quotation marks. Terms can be explicitly or implicitly defined in a specific embodiment and / or elsewhere in the application document.

[0057] As used herein, a "computer system" may include, for example, one or more servers, motherboards, processing nodes, laptops, tablets, personal computers (portable or non-portable), personal digital assistants, smart phones, smart watches, smart bracelets, cellular or mobile phones, other mobile devices with at least a processor and memory, video game systems, augmented reality systems, holographic projection systems, televisions, wearable computing systems, and / or other devices that provide one or more processors controlled at least in part by instructions. Instructions may take the form of firmware or other software in memory and / or dedicated circuit systems.

[0058] A "multithreaded" computer system is one that supports multiple threads of execution. The term "thread" should be understood to include any code that can be scheduled or subject to scheduling (and possibly synchronization), and may also be referred to by another name such as "task", "process" or "coroutine", for example. Threads may run in parallel, sequentially, or in a combination of parallel execution (e.g., multiprocessing) and sequential execution (e.g., time slicing).

[0059] A "processor" is a thread processing unit, such as a core in a simultaneous multithreading implementation. A processor comprises hardware. A given chip may house one or more processors. Processors may be general purpose, or may be customized for specific uses, such as vector processing, graphics processing, signal processing, floating point arithmetic processing, encryption, I / O processing, machine learning, etc.

[0060] The "kernel" includes the operating system, hypervisor, virtual machine, BIOS or UEFI code, and similar hardware interface software.

[0061] "Code" means processor instructions, data (including constants, variables, and data structures), or both instructions and data. "Code" and "software" are used interchangeably herein. Executable code, interpreted code, and firmware are some examples of code. Code that must be interpreted or compiled in order to execute is often referred to as "source code", but when the context clearly indicates to the technician that the code in question is source code (e.g., in the context of submitting updated code for inclusion in a repository), source code may be simply referred to as "code".

[0062] "Program" is used broadly herein to include applications, kernels, drivers, interrupt handlers, firmware, state machines, libraries, and other code written by programmers (also referred to as developers) and / or automatically generated.

[0063] “Service” means a consumable program provided in a cloud computing environment or other network or computing system environment.

[0064] "Cloud" means pooled resources for computing, storage, and networking that are elastically available for measured, on-demand services. Clouds can be private, public, community, or hybrid, and cloud services can be provided as infrastructure as a service, platform as a service, software as a service, or another service. Unless otherwise specified, any discussion of reading from or writing to files includes reading / writing local files or reading / writing over a network (which may be a cloud network or other network), or both (local and networked read / write).

[0065] "IoT" or "Internet of Things" refers to any networked collection of addressable embedded computing nodes. Such nodes are examples of computer systems as defined herein, but they also have at least two of the following characteristics: (a) no local human-readable display; (b) no local keyboard; (c) the primary source of input is a sensor that tracks a source of non-verbal data; (d) no local rotatable disk storage - a RAM chip or ROM chip provides the only local memory; (e) no CD or DVD drive; (f) embedded in household appliances; (g) embedded in implanted medical devices; (h) embedded in vehicles; (i) embedded in process automation control systems; or (j) designs focused on one of the following: environmental monitoring, municipal infrastructure monitoring, industrial equipment monitoring, energy usage monitoring, human or animal health monitoring, or physical transportation system monitoring.

[0066] As used herein, "comprising" allows for additional elements (ie, comprising means including) unless stated otherwise.

[0067] "Optimization" means improvement, not necessarily perfection. For example, it may be possible to make further improvements in a program or algorithm that has already been optimized.

[0068] For example, "process" is sometimes used herein as a term in the field of computer science and in a technical sense covers resource users, i.e., coroutines, threads, tasks, interrupt handlers, application processes, kernel processes, procedures, and object methods. "Process" is also used herein as a term in the field of patent law, for example, when describing a process claim as opposed to a system claim or an article (configured storage medium) claim. Similarly, "method" is sometimes used herein as a technical term in the field of computer science (a "routine") and as a term in the field of patent law (a "process"). Those skilled in the art will understand the meaning intended to be used in a particular case, and will also understand that a given claimed process or method (in the patent law sense) can sometimes be implemented using one or more processes or methods (in the computer science sense).

[0069] "Automatically" means utilizing automation (e.g., general-purpose computing hardware configured by software for the specific operations and technical effects discussed herein), as opposed to without automation. In particular, the steps performed "automatically" are not performed manually on paper or in a person's mind, although they may be initiated by a human or directed interactively by a human. The automated steps are performed by a machine to obtain one or more technical effects that could not be achieved without utilizing the technical interaction so provided.

[0070] Those skilled in the art understand that the technical effect is the presumed purpose of the technical embodiment. For example, in one embodiment, calculations are involved, and some calculations can also be performed without a technical component (e.g., by paper and pencil, or even as a mental step), and this fact does not remove the existence of the technical effect or change the specific and technical nature of the embodiment. Operations such as the following should be understood as inherently digital in this article: receiving source code update submissions for inclusion in a build or repository, determining the extent to which source code is covered by a test suite, marking digital artifacts such as code update submissions and test sets, and executing code. The human mind cannot be directly docked with a CPU or other processor or with a RAM or other digital storage device to read and write the necessary data to perform the software testing assurance steps taught herein. One reason for the existence of automated program testing - perhaps the main reason - is that manual testing using only mental processes or paper and pencil is actually seriously insufficient for today's commercial software development workload. In view of this disclosure, those skilled in the art will understand this well, but sometimes may need to inform or remind others of the fact. Unless otherwise stated, it is assumed that the embodiments are capable of operating at scale (ie, operating on event data from a hundred or more monitored devices) in a commercial production environment or in a test lab within a production environment, rather than simply being thought experiments.

[0071] "Computationally" also means that a computing device (at least a processor plus memory) is being used, and excludes results obtained by human thought alone or by human action alone. For example, as understood herein, arithmetic performed with paper and pencil is not computationally performing arithmetic. The computational results are faster, more extensive, deeper, more accurate, more consistent, more comprehensive, and / or otherwise provide a technical effect beyond the scope of human performance alone. A "computational step" is a step that is computationally performed. Neither "automatically" nor "computationally" necessarily means "immediately." "Computationally" and "automatically" are used interchangeably herein.

[0072] "Actively" means without direct request from the user. In fact, the user may not even be aware that active steps of the embodiments are possible until the results of the step have been presented to the user. Unless otherwise stated, any computational and / or automatic steps described herein may also be completed actively.

[0073] Throughout this document, the use of the optional plural "(s)" means that there are one or more of the indicated features. For example, "(s) processors" means "one or more processors" or equivalently "at least one processor".

[0074] For legal and practical purposes, use of the word "step" herein, in the claims or elsewhere, is not intended to invoke means-plus-function or step-plus-function. Any presumption to such effect is hereby expressly rebutted.

[0075] For legal and practical purposes, unless the claims use the phrase “means for,” they are not intended to invoke a parts-plus-function interpretation. Claim language that is intended to be interpreted as parts-plus-function language, if any, will expressly recite that intent through the use of the phrase “means for.” When a parts-plus-function interpretation is employed, whether through the use of “means for” and / or through legal interpretation of the claim language by a court, the parts listed in the specification for a given noun or a given verb should be understood to be linked together with the claim language and are linked together here in any of the following ways: appearing in the same box in a drawing block diagram, being represented by the same or similar name, being represented by the same figure number, describing a functional relationship in any drawing, and indicating a functional relationship in the text of this disclosure. For example, if a claim limitation recites a "zac widget" and that claim limitation is subject to a parts-plus-function interpretation, then all structures identified at least anywhere that "zac widget" is mentioned in any figure, paragraph, or example in the specification or tied together by any figure markings assigned to the zac widget, or disclosed as having a functional relationship to the structure or operation of the zac widget, will be considered part of the structure identified in applications of the zac widget and will help define an equivalent set of zac widget structures.

[0076] Throughout this document, unless otherwise explicitly stated, any reference to a step in a process assumes that the step may be performed directly by an interested party, and / or indirectly by a party through an intervening mechanism and / or intervening entity, and still fall within the scope of the step. That is, unless direct performance is an explicitly stated requirement, the step need not be performed directly by the interested party. For example, a step involving an action on a destination or other subject matter by an interested party (such as accept, ascertain, assert, assure, construct, check, communicate, cover, determine, discern, display, perform, indicate, generate, call, notify, pull, read, receive, report, reside, test, update, warn (and accept, be accepted, ascertain, be ascertained, etc.)) may involve an intervening action by another party (such as forwarding, copying, uploading, downloading, encoding, decoding, compressing, decompressing, encrypting, decrypting, authenticating, calling, etc.), but should still be understood to be performed directly by the interested party.

[0077] Whenever reference is made to data or instructions, it should be understood that, for example, these items configure computer-readable memory and / or computer-readable storage media, thereby converting it into a specific article, as opposed to simply existing on paper, in one's mind, or as a signal propagating on a wire. For purposes of patent protection, memory or other computer-readable storage media are not propagating signals, carrier waves, or energy alone, which are outside the scope of patentable subject matter. No claim covers signals per se or energy alone, and any claim interpretation asserting the contrary is unreasonable in light of this disclosure. Unless otherwise expressly stated in a granted claim, a claim does not cover signals per se or energy alone.

[0078] Furthermore, despite apparent provisions to the contrary elsewhere herein, a distinct distinction should be understood between (a) computer-readable storage media and computer-readable memory on the one hand and (b) transmission media (also known as signal media) on the other hand. Transmission media are propagating signal or carrier wave computer-readable media. In contrast, computer-readable storage media and computer-readable memory are not propagating signal or carrier wave computer-readable media. Unless otherwise expressly stated in the claims, "computer-readable medium" means computer-readable storage media, not propagating signals per se, nor energy alone.

[0079] The "embodiments" herein are examples. The term "embodiments" is not interchangeable with "the present invention." The embodiments may freely share or borrow aspects to create other embodiments (as long as the result is operable), even if the resulting combination of aspects is not explicitly described herein itself. It is not necessary for a person skilled in the art to require each permitted combination to be explicitly and individually described, which is contrary to the policy of recognizing that patent specifications are written for those skilled in the art. Formal combinatorial calculations and informal intuition about the number of possible combinations resulting from even a small number of combinable features will show that there are a large number of aspect combinations of the aspects described herein. Therefore, requiring an explicit description of each and every combination would run counter to the policy of requiring concise patent specifications and making the reader knowledgeable about the relevant technical field.

[0080] List of reference numerals

[0081] The following list is provided for the convenience and support of the drawings and as part of the text of the specification, which describes the innovation by reference to multiple items. Nevertheless, items not listed here may still be part of a given embodiment. In order to make the text clearer and easier to read, given reference numerals are described near some (but not all) referenced items in the text. The same reference numerals may be used to refer to different examples or different instances of a given item. The list of reference numerals is:

[0082] 100: Operating environment, also known as computing environment

[0083] 102: Computer system, also called computing system or computing system

[0084] 104: User

[0085] 106: Peripheral devices

[0086] 108: Networks, typically including LANs, WANs, software-defined networks, and other wired or wireless networks

[0087] 110: Processor

[0088] 112: Computer-readable storage media, such as RAM, hard disk

[0089] 114: Removable configured computer readable storage medium

[0090] 116: instructions, executable by a processor; may be on a removable storage medium, or in other memory (volatile or non-volatile or both)

[0091] 118: Data

[0092] 120: (multiple) kernels, e.g. (multiple) operating systems, BIOS, UEFI, device drivers

[0093] 122: Tools, for example, antivirus software, firewalls, packet sniffer software, intrusion detection systems (IDS), intrusion prevention systems (IPS), debuggers, profilers, compilers, interpreters, static code analysis tools, software development tools and tool kits, hardware development tools and tool kits, diagnostics

[0094] 124: Applications, e.g., word processors, web browsers, spreadsheets, games, email tools

[0095] 126: Display screen, also known as "monitor"

[0096] 128: computing hardware not otherwise associated with reference numerals 106, 108, 110, 112, 114

[0097] 202: Computing systems enhanced with software test assurance techniques

[0098] 204: Software Testing Ensures Code

[0099] 206: Software testing ensures the user interface of the code

[0100] 208: tool chain, e.g., one or more software development tools 122 that can collectively convert source code into executable code

[0101] 210: Integrated Development Environment

[0102] 212: Extensions to Extensible Development Tools

[0103] 214: Extensible development tools, such as Visual Tools, Visual Code Tools (a brand of Microsoft Corp.), Tools (a brand of Apple Computer, Inc.), Studio Tools (a brand of Google, LLC), etc.

[0104] 216: A commit that contains updated source code (either directly as a copy or indirectly via a pin, index, or other reference); may also contain other data such as timestamps, digital signatures, author names, and annotations; some examples of commits include (a) pull requests, (b) commits to a source code version control system, whether using HTTP, HTTPS, or another network protocol, (c) commits to a source code version control system that does not include a peer review and comment step, and (d) commits to a tool chain

[0105] 218: The portion of commit 216 that contains the updated source code; may also include adjacent or surrounding code that has not been updated

[0106] 220: Source code

[0107] 222: Update to source code; may take the form of text insertion, text deletion, text replacement, or a mixture thereof; unless otherwise indicated, an "update" is a set of source code lines that embody a text insertion, text deletion, or text replacement, even if not every character of the line in question has been changed from a previous version of the source code, but does not include lines of source code in which no changes have been made

[0108] 224: Acceptance status, e.g., accepted / rejected for peer review, accepted / rejected for inclusion in a repository, or accepted / rejected for inclusion in a build;

[0109] 226: Flag, e.g., bit, enumeration value, integer value, or other discrete value indicating, e.g., that a test guarantees the presence, absence, or is under review

[0110] 228: Items related to test assurance, such as code update submissions 216, submitted code portions 218, or tests or test sets 232 used to test the code

[0111] 230: Test suites that can be used for automated testing of software

[0112] 232: Test set, i.e., one or more specific tests within a test suite

[0113] 234: Test coverage reporting tools, such as Coverage.py, EMMA, JaCoCo, Serenity BDD, Clover TM (a brand of Atlassian Pty Ltd), BullseyeCoverage, CoCo TM (brand of FrogLogic GmbH), or another commercially available tool that reports test coverage

[0114] 236: Test Coverage Report

[0115] 238: Automated test management tools, e.g. (a brand of QASymphony LLC), (a brand of HSPractiTest LLC), (a brand of Inflectra Corporation), (a brand of Panaya Ltd.), or another commercially available tool that assists in the management of automated testing

[0116] 240: Generally, computing system hardware, such as processor(s) 110, memory 112, screen(s) 126, other hardware 128, and other hardware not specifically numbered but still taught herein

[0117] 242: Notice or warning

[0118] 244: Timestamp

[0119] 300: Types of code coverage

[0120] 302: Code coverage through automated testing

[0121] 304: Function coverage, that is, the extent to which each function or other subroutine in the code has been called

[0122] 306: Decision coverage, i.e., the extent to which entry and exit points in the code have been called at least once, and which decisions in the code have been made at least once for all possible outcomes 308: Statement coverage, i.e., the extent to which each statement in the code has been executed

[0123] 310: Finite state machine coverage, that is, the extent to which each state of the finite state machine implemented in the code has been reached

[0124] 312: Branch coverage, that is, the extent to which each branch in a control structure in the code has been executed; examples of control structures include if-then statements, switch statements or case statements, and jump tables

[0125] 314: Entry / exit coverage, i.e., the extent to which each entry (e.g., call) and exit (e.g., return) of a subroutine in the code has been executed

[0126] 316: Path coverage, that is, the extent to which each path through the code has been executed

[0127] 318: Data flow coverage, that is, the extent to which each variable definition and its usage has been reached

[0128] 320: Condition coverage, that is, the extent to which Boolean subexpressions in the code have been evaluated to both true and false

[0129] 322: Loop coverage, that is, the extent to which each loop has been executed zero, one, or more than one times during the execution of the code

[0130] 400: Scenarios illustrating different ways that source code 220 and tests 232 may be handled in conjunction with bug fixes and feature changes

[0131] 700: Software testing assurance method; also a flowchart illustrating such a method

[0132] 702: Receive code update submission

[0133] 704: Receive the updated code part of the code update submission

[0134] 706: Determine whether the updated source code portion 218 is at least partially covered by the test set

[0135] 708: Find out if the test set has changed

[0136] 710: Identify the source code impact areas of changes in the test set

[0137] 712: Impact area of ​​changes in the test set

[0138] 714: Marks one or more items; also called setting a tag

[0139] 716: Indicates that there is a lack of assurance that adequate test coverage exists

[0140] 718: Avoid labeling any item; also known as avoiding setting labels

[0141] 800: Flowchart

[0142] 802: Ensure software testing is optimized

[0143] 804: Software testing (automated unless otherwise specified)

[0144] 806: Using software testing to cover code

[0145] 808: Accept updated source code for one or more development activities

[0146] 810: Include source code in repository

[0147] 814: Peer Review of Source Code

[0148] 816: Using source code to build executables

[0149] 820: Generate code coverage report

[0150] 822: Check code coverage through one or more test suites

[0151] 824: Read coverage report

[0152] 826: Assert a value or equality of values ​​as part of a code test

[0153] 828: Receiving pull requests

[0154] 830: Pull Request

[0155] 832: Invoke a tool, such as a coverage reporting tool or other tool

[0156] 834: Digitally communicate with a tool (e.g., a test management tool or other tool)

[0157] 836: Display a notification or an alert, for example, on the screen or in a listing or printout, or place a notification or an alert in a file, or transmit a notification or an alert; also the action known as notification or alert

[0158] 838: Resides in a software development tool chain, e.g., within a tool in the tool chain

[0159] 840: Check timestamp

[0160] 842: Identify the relevance of tests on one or more lines of source code

[0161] 844: Reject submitted code

[0162] 902: Repository, e.g., a repository that supports collaboration between developers and provides version control; a repository that provides services for hosting source code and development projects. Some of the most widely used web-based repositories include (a brand of GitHub, Inc.), (a brand of Atlassian Pty Ltd) and (A brand of SourceForge Media, LLC)

[0163] Operating Environment

[0164] refer to Figure 1 , an operating environment 100 for an embodiment includes at least one computer system 102. Computer system 102 may or may not be a multiprocessor computer system. The operating environment may include one or more machines in a given computer system, which may be clustered in a cloud, client-server networked, and / or peer-to-peer networked. An individual machine is a computer system, as is a group of cooperating machines. A given computer system 102 may be configured for, for example, an end user with an application, for an administrator, as a server, as a distributed processing node, and / or in other ways.

[0165] Human users 104 can interact with computer system 102 via typed text, touch, voice, movement, computer vision, gestures, and / or other forms of I / O using displays, keyboards, and other peripherals 106. Screen 126 can be a removable peripheral 106, or can be an integral part of system 102. A user interface can support interaction between an embodiment and one or more human users. The user interface can include a command line interface, a graphical user interface (GUI), a natural user interface (NUI), a voice command interface, and / or other user interface (UI) presentations, which can be presented as different options or can be integrated.

[0166] System administrators, network administrators, cloud administrators, security personnel, operations personnel, developers, testers, engineers, auditors, and end users are all specific types of users 104. Automated agents, scripts, playback software, etc. that represent one or more individuals can also be users 104, for example, to support testing system 102. In some embodiments, storage devices and / or networking devices can be considered peripheral devices, while in other embodiments, they can be considered part of system 102, depending on their separability from processor 110. For example, Figure 1 Other computer systems not shown may interact with computer system 102 or with another system embodiment in a technical manner using one or more connections to network 108 via a network interface device.

[0167] Each computer system 102 includes at least one processor 110. As with other suitable systems, the computer system 102 also includes one or more computer-readable storage media 112. The storage media 112 can be of different physical types. The storage media 112 can be volatile memory, non-volatile memory, local fixed media, removable media, magnetic media, optical media, solid-state media, and / or other types of physical persistent storage media (as opposed to only propagating signals or only energy). In particular, when inserted or otherwise installed, a configured storage medium 114 (such as a portable (i.e., external) hard drive, CD, DVD, memory stick, or other removable non-volatile storage medium) can functionally become a technical part of the computer system, making its content accessible for interaction with the processor 110 and use by the processor 110. Removable configured storage media 114 is an example of a computer-readable storage medium 112. Some other examples of computer-readable storage media 112 include built-in RAM, ROM, hard disk, and other memory storage devices that are not easily removable at any time by the user 104. For purposes of the present patent requirements, under any claim pending or approved, neither the computer-readable medium, computer-readable storage medium, nor the computer-readable memory is a signal per se, or merely energy.

[0168] For example, the storage medium 114 is configured with binary instructions 116 executable by the processor 110; "executable" is used in a broad sense herein to include, for example, machine code, interpretable code, byte code, and / or code running on a virtual machine. The storage medium 114 is also configured with data 118, which is created, modified, referenced, and / or otherwise used for technical effect by the execution of the instructions 116. The instructions 116 and data 118 configure the memory or other storage medium 114 in which they reside; when the memory or other computer-readable storage medium is a functional part of a given computer system, the instructions 116 and data 118 also configure the computer system. In some embodiments, portions of the data 118 represent real-world items, such as product characteristics, inventory, physical measurements, settings, images, readings, targets, volumes, and the like. Such data is also converted through backup, restore, commit, abort, reformat, and / or other technical operations.

[0169] A given operating environment 100 may include an integrated development environment (IDE) 210 that provides developers with a coordinated set of computing technology development tools, such as compilers, source code editors, profilers, debuggers, layout tools, simulators, etc. In particular, some suitable operating environments for some software development embodiments include or help create a program that is configured to support program development. Visual Development Environment (a brand of Microsoft Corporation). Some suitable operating environments include environment (a brand of Oracle America, Inc.), and some suitable operating environments include environments that utilize languages ​​such as C++ or C# ("C-Sharp"), but many of the teachings herein are applicable to a variety of programming languages, programs, programming models, development tools, and development methodologies.

[0170] Although the embodiments may be described as being implemented by software instructions executed by one or more processors in a computing device (e.g., a general-purpose computer, a server, or a cluster), this description is not meant to exhaust all possible embodiments. It will be appreciated by those skilled in the art that the same or similar functionality may also be implemented in whole or in part directly in hardware logic to provide the same or similar technical effects. Alternatively, or in addition to software implementations, the technical functionality described herein may be at least partially performed by one or more hardware logic components. For example, and without excluding other implementations, the embodiments may include hardware logic components 110, 128, such as field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on chip components (SOCs), complex programmable logic devices (CPLDs), and similar components. For example, the components of the embodiments may be aggregated into multiple interactive functional modules based on the inputs, outputs, and / or their technical effects of the components of the embodiments.

[0171] In addition to the processor 110 (e.g., CPU, ALU, FPU and / or GPU), memory / storage medium 112, and display 126, the operating environment may also include other hardware 128, such as, for example, batteries, buses, power supplies, wired and wireless network interface cards. The terms "screen" and "display" are used interchangeably herein. The display 126 may include one or more touch screens, a screen that responds to input from a pen or tablet computer, or a screen that operates only for output. In some embodiments, peripheral devices 106 such as human user I / O devices (screens, keyboards, mice, tablet computers, microphones, speakers, motion sensors, etc.) will be presented as being in operable communication with one or more processors 110 and memory.

[0172] In some embodiments, the system includes multiple computers connected by a wired or wireless network 108. The networking interface device 128 can provide access to the network 108 using, for example, network components that may be present in a given computer system (e.g., a packet switching network interface card, a wireless transceiver, or a telephone network interface). Virtualization of the networking interface device and other network components such as switches or routers or firewalls can also exist in, for example, a software defined network or sandboxed or other secure cloud computing environment. A given embodiment may also transmit technical data and / or technical instructions via direct memory access, removable non-volatile storage media, or other information storage-retrieval and / or transmission methods.

[0173] Those skilled in the art will appreciate that the foregoing aspects and other aspects presented herein in the context of an "operating environment" may form part of a given embodiment. The headings of this document are not intended to strictly categorize features into embodiment and non-embodiment feature sets.

[0174] One or more items are shown in outline form in the figures, or listed in brackets, to emphasize that they are not necessarily part of the operating environment shown or part of all embodiments, but may interoperate with items in the operating environment or some embodiments discussed herein. In any figure or any embodiment, items that do not follow the non-outline or bracket form are required. In particular, Figure 1 It is provided for convenience; Figure 1 The inclusion of an item does not imply that the item or the described use of the item was known prior to the present invention.

[0175] More information about the system

[0176] Examples are provided herein to help illustrate various aspects of the technology, but the examples given in this document do not describe all possible embodiments. The embodiments are not limited to the specific implementations, arrangements, displays, features, methods, or scenarios provided herein. A given embodiment may include, for example, additional or different technical features, mechanisms, sequences, data structures, or functionality, and may otherwise depart from the examples provided herein.

[0177] Figure 1-Figure 6 The diagram illustrates aspects of some architectures suitable for use with the embodiments taught herein. Figure 2 , the system 102 can be enhanced with one or more software test assurance functionalities described herein to produce or utilize a test assurance enhanced system 202. The enhanced system 202 includes at least test assurance code 204, which operates according to at least one example provided herein. The assurance code 204 includes or communicates with a user interface 206; GUIs and other example user interface technologies are discussed elsewhere herein. For example, the assurance code 204 can reside in a development tool chain 208, an integrated development environment 210, a development tool 214, or a development tool extension 212. The skilled person will understand that items 208, 210, 212, 214, and 122 are not necessarily mutually exclusive, for example, a given tool 122 can be a development tool 214 in an IDE 210.

[0178] In this example, the enhanced system 202 receives or otherwise includes a code update submission 216 that may include different parts (e.g., a routing part, an authorization part, a history part, a timestamp 244, etc.). In particular, the code update submission 215 has a portion 218 that directly or indirectly includes source code 220. At least a portion of the source code is updated relative to a previous portion of the source code; the primary role of the code update submission 216 is to submit updates 222 for inclusion in the body of the source code. In practice, some submitted source code may not have associated tests 232, but the present disclosure is particularly interested in situations involving both updated source code and associated test sets 232 from a test suite 230 that can be used to automatically test at least a portion of the submitted source code in some manner to some degree.

[0179] In this example, the acceptance status 224 of the submission 216 with respect to one or more workflow states can be tracked in a variable or other data structure. For example, a given status 224 may indicate that the submission has been received and queued for analysis, or that the submission is undergoing code coverage analysis to discern which portions of the associated test set (if any) 232 are related to the changes 222 that will be made in the source code by fully accepting the submission. More generally, the acceptance status 224 can indicate any location in any workflow or other process discussed herein. When the workflow concludes that the test set 232 associated with the update 222 has itself not been updated, a flag 226 indicating this conclusion can be set on one or more items 228, for example, the flag can be set on the submission itself, on the source code in the submission, on the test set, or a combination thereof.

[0180] In this example, the enhanced system 202 includes or communicates with a coverage reporting tool 234 that generates a coverage report 236 that indicates the extent to which the submitted source code 220 is covered by the associated test suite tests 232. The coverage report may simply indicate which line or lines of code are covered by the tests. However, in some embodiments, the coverage report also indicates or alternatively indicates which categories 300 of code coverage 302 are provided by the tests 232 associated with the submitted code.

[0181] Figure 3Several categories of code coverage 302 are illustrated, which are defined in the list of reference numerals, including function coverage 304, decision coverage 306, statement coverage 308, finite state machine coverage 310, branch coverage 312, entry / exit coverage 314, path coverage 316, data flow coverage 318, condition coverage 320, and loop coverage 322. In situations where code coverage for a single category of testing discussed herein is indicated, but which category 300 is not explicitly stated, statement coverage 308 is assumed.

[0182] In this example, the enhanced system 202 also communicates with or includes an automated test management tool 238. Figure 2 The test management tool 238 and the test reporting tool 234 are depicted in different blocks, but those functionalities may be combined into a single tool 122 or coordinated modules in a given tool chain 208, for example.

[0183] Figure 2 Also shown is a notification 242. Due to lack of space, Figure 2 Warning 242 is also not explicitly shown, but it may also be present in the assurance code 204 of the enhanced system 202 or displayed by the enhanced system 202. Notifications provide the user with a conclusion or status about the input, for example, a notification may report an acceptance status or mark. Notifications are not necessarily negative. Instead, a warning is a notification that indicates an error, or that an undesirable constraint on development exists or is believed to exist. For example, a warning may tell the user that a submitted code update 222 will not be pulled into the repository until an updated test 232 is submitted and executed against the code update 222.

[0184] Figure 4-Figure 6 Several scenarios 400 are illustrated in which source code 220 and associated test sets 232 may be updated to some extent, ranging from not at all to substantially updated to completely updated. Other scenarios are possible; the examples discussed herein are not comprehensive. Moreover, while some embodiments require test updates in some situations for acceptance of code updates, in general, updates to test sets 232 are not necessarily mandatory under every embodiment presented herein, neither in the sense that test updates of a certain kind are mandatory, nor in the sense that test updates that provide code coverage 302 of a particular kind 300 are mandatory, nor in the sense that test updates that provide code coverage 302 of a particular degree are mandatory.

[0185] For example, those skilled in the art of project management will recognize that updating of the test set is one possible technical effect of the embodiment, but not the only possible technical effect. Informed acceptance of technical risk is another technical effect, for example, a developer may be notified of a risk detected where a test update was prudent in light of a source code update, but then the developer decides to accept the risk that a bug has been introduced or remains in the updated code, and chooses to continue to utilize the updated code despite being aware of the technical risk.

[0186] Considering the foregoing context, Figure 4 Seven scenarios 400 are illustrated, denoted for reference as Scenario A through Scenario G. These scenarios are not listed in any particular order, and as noted above, are not presented as a comprehensive collection of all scenarios involving updates to source code, or updates to associated tests for that source code, or both. Figure 7 The diagram shows Figure 4 The same seven scenarios, but using idealized source code. The idealized source code is depicted graphically using points and lines rather than source code text to emphasize that this discussion is not limited to any particular programming language.

[0187] Figure 5 An idealized portion of source code 220 is shown to illustrate different ways in which source code can be updated. For ease of reference, Figure 5 The rows in which the idealized source code is listed are numbered (1) to (4), but the teachings herein also apply to source code that is not so arranged in columns of rows. Figure 5 The left side shows the source code before the edit that produced the update, and Figure 5 The right side of shows the code after edits have been made. The source code in line (1) on the left has not been edited; it is the same in the updated portion of the code on the right as it was in the previous version of the code on the left. The code in line (2) on the left has also not been edited. The code in line (3) on the left occupies line (4) on the right because a new line of code has been inserted above it. The new line of code is represented on the right as four consecutive dots. The number of points used in this figure is not important to the teachings presented herein. The source code in line (3) on the left has been edited by inserting text, deleting text, replacing text, or some combination thereof. The edited code is represented on the right as seven consecutive dots. Again, the number of points used in this figure is not important to the teachings presented herein. The code in line (4) on the left has been deleted by editing, so it has no corresponding point or line segment representing it on the right. In short, as Figure 5As shown, updates to source code may be implemented using text insertion, text deletion, text replacement, or a combination thereof, and the updates do not necessarily change every line of code in the submitted code portion 218 .

[0188] Given that Figure 4 and Figure 5 , Figure 6 It can be understood as illustrating the following scenarios. In each scenario, only a portion of the source code (depicted as a point) has been updated. In scenario A, the changed tests provide at least some coverage of the entire submitted portion 218 of the source code. In scenario B, the entire submitted portion 218 of the source code is again covered by tests, but those tests have not been updated. In scenario C, the updated tests provide at least some coverage of the updated portion of the submitted portion 218 of the source code; no tests cover the non-updated portion of the submitted code. In scenario D, the non-updated tests provide at least some coverage of the updated portion of the submitted portion 218 of the source code; no tests cover the non-updated portion of the submitted code. In scenario E, the tests are updated, but they only cover the non-updated portion of the source code. In scenario F, the non-updated tests cover the non-updated portion of the source code; no tests cover the updated source code. In scenario G, no tests cover any part of the submitted source code portion.

[0189] In some embodiments, one or more of scenarios A through G are detected, and then in each scenario except A and C, the embodiment notifies the user that it would be prudent to update the test suite 230 to cover at least the updated portion of the submitted code. In scenarios A and C, the embodiment determines that some updates to the tests related to the updates in the submitted source code have been made, and therefore no warning is given.

[0190] Some embodiments use or provide a computing system 202, which is operable to support software test assurance. The example system includes a processor 110, a memory 112 in operable communication with the processor, and an assurance code 204. Upon execution by the processor, the assurance code 204 performs a software test assurance method, which includes receiving a code update submission 216, which is represented here as "S-current". The submission "S-current" includes a portion 218, which is represented here as "P-current", which in turn includes an update 222 to the source code 220. The software test assurance method also includes determining that the portion "P-current" is at least partially covered by a test set 232 (represented here as "TS-current") of a test suite 230. The software test assurance method also includes ascertaining whether the test set "TS-current" is different from the test set "TS-previous", and the "TS-previous" at least partially covers the portion "P-previous" of the code update submission "S-previous". In this context, "S-Previous" is older than "S-Current", and "P-Current" updates "P-Previous". The software test assurance method also includes: when the ascertainment ascertains that "TS-Current" is not different from "TS-Previous", marking at least one item 228 based on the "TS-Current" test set to indicate a lack of test assurance for "P-Current" update. The marking sets a flag of at least one of the following: "S-Current", "P-Current" or "TS-Current".

[0191] In some embodiments, system 202 includes an integrated development environment 210 and ensures that code 204 resides at least partially in the integrated development environment.

[0192] In some embodiments, system 202 includes an extensible software development tool 214, and the guarantee code 204 resides at least partially in an extension 212 of the extensible software development tool. When the extension 212 is functionally installed in the extensible software development tool, the guarantee code 204 also resides in the extensible software development tool accordingly.

[0193] In some embodiments, a test set 232 "TS-Current" resides in the system 202, and the test set "TS-Current" includes assertions of values ​​that will result from the execution of at least a portion of "P-Current". As an example of such an assertion, the PQ() function detailed elsewhere herein accepts an integer value representing a month and returns a string indicating which calendar quarter the month is in; "PQ" stands for "Print Quarter". One of the associated assertion tests 232 asserts that PQ(2) is equal to "First Quarter" because 2 represents February, and February is in the first quarter of the calendar year.

[0194] In some embodiments, system 202 is configured to require that any flag 226 set by assurance code 204 be resolved before part "P-current" is accepted. This naturally leads to the question of what accepted means. In some embodiments, part 218 being received means at least one of the following: the part is received for inclusion in a repository, the part is accepted for peer review, or the part is accepted for inclusion in a build.

[0195] In some embodiments, the system 202 also includes a coverage report 236 generated by a coverage reporting tool 234. The coverage report represents the test coverage 302 of at least a portion of the source code by the test set "TS-current". In some embodiments, the system 202 also includes or alternatively includes the coverage reporting tool 234 itself. The coverage reporting tool 234 is executable to check the test coverage 302 of at least a portion 218 of the source code 220 by the test set 232 "TS-current".

[0196] Other system embodiments are also described herein, either directly or as system versions of the described methods or configured media, which are informed by the expanded discussion of computing hardware herein.

[0197] method

[0198] Figure 7 A method 700 for optimizing software test assurance is illustrated. An enhanced system or other device or location receives 702 a code update submission, the code update submission specifically including: receiving 704 a portion of the submission containing some source code, at least a portion of which is updated source code. The illustrated method determines 706 that the source code portion is at least partially covered by a test suite. For example, this may include determining Figure 6 One of the scenarios A to F of the embodiment applies, as opposed to scenario G. The illustrated method finds out whether the relevant test set is different from the previous version, in other words, whether the test set related to the source code portion has changed. This may include identifying the impact area 712 of the test set difference or being associated with it. Figure 6 Several different impact areas are illustrated in dashed lines, such as Figure 6 As indicated in the legend, it associates the area enclosed in the dashed line with "changed test coverage". Returning attention to Figure 7 , the illustrated method either marks 714 the item(s) to indicate 716 a lack of testing assurance in the specific context defined by the portion, update, and test set version, or otherwise the illustrated method avoids 718 setting such a mark.

[0199] Unless otherwise indicated, the technical methods shown in the figures or otherwise disclosed are automatically performed by tools enhanced, for example, with assurance code 204. To the extent that human administrators or other humans are involved in the actions, the methods may also be performed partially automatically and partially manually, for example, entering commands to submit portions of source code to a repository or building a project executable using updated source code. None of the methods contemplated as innovative herein are entirely manual. In a given embodiment, zero or more of the illustrated steps of a method may be repeated, perhaps with different parameters or data. The steps in the embodiments may also be performed in accordance with the same procedures as described above. Figure 7 and Figure 8 The steps can be performed in a different order from the order from top to bottom shown in . The steps can be performed sequentially, in a partially overlapping manner or completely in parallel. In particular, the order of the steps performed during the method can be changed from one execution of the method to another execution of the method by the flow chart 800 items. The traversal order can also change from one method embodiment to another method embodiment. The steps can also be omitted, combined, renamed, recombined, performed on one or more machines, or otherwise separated from the shown process, as long as the method performed is operable and meets at least one claim.

[0200] Some embodiments use or provide a method for software test assurance. The method includes electronically receiving 702 a code update submission "S-current", the code update submission "S-current" includes a portion "P-current", the portion "P-current" has one or more lines of source code, and the portion "P-current" includes an update to the source code. The method also includes automatically determining 706 that the portion "P-current" of the source code is at least partially covered by a test set "TS-current" of a test suite. The method also includes automatically finding out 708 whether the test set "TS-current" is different from the test set "TS-previous", and "TS-previous" at least partially covers the portion "P-previous" of the code update submission "S-previous", wherein "S-previous" is older than "S-current", and wherein "P-current" updates "P-previous". The method also includes automatically marking 714 at least one item based on the “TS-current” test set to indicate a lack of test assurance for updates contained in “P-current” when the determination ascertains that “TS-current” is not different from “TS-previous”, and otherwise avoiding 718 marking any item on the basis that “TS-current” is not different from “TS-previous” relative to “P-current”, wherein the at least one marked item includes at least one of the following: “S-current”, “P-current”, or “TS-current”.

[0201] In some embodiments, where the ascertainment ascertains that "TS-current" is different from "TS-previous", the method further includes identifying 842 that there are no differences between "TS-previous" and "TS-current" with respect to any tests covering "P-current", and then marking 714 the item to indicate a lack of test assurance for updates included in "P-current".

[0202] In some embodiments, where the ascertainment ascertains that "TS-current" is different from "TS-previous", the method further includes identifying 842 that there are no differences between "TS-previous" and "TS-current" related to any tests covering updates in "P-current", and then marking 714 the item to indicate a lack of test assurance for updates included in "P-current".

[0203] In some embodiments, receiving 702 a code update commit "S-current" includes receiving 828, 702 a pull request 830, i.e., a data structure including a request for a portion "P-current" to be pulled into a repository 902, the repository 902 containing source code for a project, application, program, or other artifact that relies on source code compilation or interpretation to provide functionality.

[0204] In some embodiments, determining 706 that the portion “P-current” of the source code is at least partially covered by the test set “TS-current” includes reading 824 a coverage report generated by a coverage reporting tool, for example, by opening a file containing the report and parsing the contents of the file, or by receiving and parsing an XML data structure or another digital data structure containing information about test coverage 302. In some embodiments, determining 706 that the portion “P-current” of the source code is at least partially covered by the test set “TS-current” includes invoking 832 a coverage reporting tool and communicating 834 with the coverage reporting tool to check coverage of at least a portion of the source code by the test set “TS-current”.

[0205] In some embodiments, ascertaining 708 whether the test set "TS-current" is different from the test set "TS-previous" includes checking 840 the timestamp of the test set "TS-current". In some, ascertaining 708 includes invoking 832 an automated test management tool and communicating 834 with the automated management tool, which tracks at least the test set "TS-current".

[0206] In some embodiments, where at least one item is marked, the method further includes notifying 836 a user by displaying a notification 242 indicating that the line of source code is at least partially covered by a test, but that it should be reviewed because the test does not necessarily provide complete coverage. In some embodiments, the method includes displaying 836 a notification 242 listing at least two of the following categories of code coverage 302: function coverage 304, statement coverage 308, branch coverage 312, condition coverage 320, decision coverage 306, path coverage 316, entry / exit coverage 314, loop coverage 322, finite state machine state coverage 310, or data flow coverage 318. Other embodiments display appropriate subsets of these categories of code coverage, for example, only statement coverage and branch coverage, or only function coverage alone.

[0207] Configured storage media

[0208] Some embodiments include a configured computer-readable storage medium 112. The storage medium 112 may include a disk (magnetic, optical, etc.), RAM, EEPROM or other ROM, and / or other configurable memory, particularly including a computer-readable storage medium (not just a propagating signal). In particular, the configured storage medium may be a removable storage medium 114, such as a CD, DVD, or flash memory. General purpose memory (which may be removable or non-removable, and may be volatile or non-volatile) may be configured into an embodiment using items in the form of data 118 and instructions 116 (such as test assurance code 204, acceptance status data structure 224, markup 226, coverage report 236, and tools 234, 238) to form a configured storage medium, which is read from the removable storage medium 114 and / or another source (such as a network connection). The configured storage medium 112 enables a computer system to perform the steps disclosed herein for software test assurance to detect potential deficiencies in test coverage. Thus, the figures help illustrate configured storage medium embodiments and process embodiments, as well as system and process embodiments. In particular, Figure 7 or Figure 8 Any of the process steps shown in or otherwise taught herein may be used to help configure a storage medium to form a configured storage medium embodiment.

[0209] Some embodiments use or provide a storage medium 112, 114 configured with code that, upon execution by one or more processors, performs a software test assurance method, the software test assurance method comprising: receiving 702 a code update submission "S-current", the code update submission "S-current" comprising a portion "P-current", the portion "P-current" comprising updated source code; determining 706 that the portion "P-current" is at least partially covered by a test set "TS-current" of a test suite; ascertaining 708 whether the test set "TS-current" is different from the test set "TS-previous", the "TS-previous" at least partially covering the code update submission "P-previous" is part of "S-previous", wherein "S-previous" is older than "S-current", and wherein "P-current" updates "P-previous"; and when the finding ascertains that "TS-current" is not different from "TS-previous", marking 714 at least one item based on the "TS-current" test set to indicate the lack of test assurance for "P-current", and otherwise avoiding 718 marking any item on the basis that "TS-current" is not different from "TS-previous" relative to "P-current", wherein the at least one item marked includes at least one of the following: "S-current", "P-current", or "TS-current".

[0210] In some embodiments, receiving 702 a code update commit "S-current" includes receiving 828 a pull request including a request for a portion "P-current" to be pulled into the repository.

[0211] In some embodiments, where the ascertainment ascertains 708 that "TS-current" is different from "TS-previous", the method further includes identifying 842 that there are no differences between "TS-previous" and "TS-current" with respect to any tests covering any portion of "P-current" 218, and then marking 714 the item to indicate a lack of test assurance for updates included in "P-current".

[0212] In some embodiments, where the ascertainment ascertains 708 that "TS-Previous" is different from "TS-Previous", the method further includes identifying 842 that there are no differences between "TS-Previous" and "TS-Current" related to any tests covering the update 222 in "P-Current", and then marking 714 the item to indicate a lack of test assurance for the update included in "P-Current".

[0213] In some embodiments, determining 706 that the portion of the source code “P-current” is at least partially covered by the test set “TS-current” includes reading 824 a report generated 820 by a coverage reporting tool.

[0214] In some embodiments, ascertaining 708 whether the test set TS-current is different from the test set TS-previous comprises at least one of checking 840 a timestamp of the test set TS-current, or communicating 834 with an automated test management tool.

[0215] In some embodiments, at least one item is marked 714, and the method also includes notifying 836 the user by displaying a notification indicating that at least a portion of the "P-current" source code is at least partially covered by tests, but because the tests do not necessarily provide complete coverage, they should be reviewed.

[0216] Additional Examples and Observations

[0217] Those skilled in the art will recognize that not every part of this disclosure or any specific detail therein is required to meet legal standards such as enabling, written description, or best mode. Moreover, embodiments are not limited to the specific programming languages, tool contexts, identifiers, fields, class definitions, or other implementation choices described herein. Any apparent conflict with any other patent disclosure, even from the owner of the present invention, does not interpret the effect of the claims set forth in this patent disclosure. Based on the understanding that all parts of this disclosure are relevant, some additional examples and observations are provided.

[0218] One insight that drives at least some of the embodiments is that code coverage during testing is not always equal to test coverage of the code. Therefore, for the best, possibly maximum test assurance, each bug fix pull request 830 should include test 232 changes. Test changes can be new tests or modifications to existing tests to cover the execution scenarios addressed by the bug fix. The basic assumption is that there is a test hole (i.e., a lack of testing, also referred to as test missing) because the bug was missed in earlier testing. Even if the product code being modified as part of the bug fix is ​​revealed to be covered using existing tests, new testing or modifications to the existing test set are also cautious and may be required to cover the execution scenarios.

[0219] The following code and test list illustrate instances in which lines of code are covered, but they do not necessarily cover all execution scenarios. The list is line numbered for reference, but may not have such line numbers in practice. In this example list, lines 02-15 can be used as examples of the submitted code portion 218, and lines 01, 16-26 can be used as examples of the test set 232. The test at line 20 is at least related to lines 06-07 of code; it can also be understood as being related to line 04, or even to line 02. Similarly, the test at line 21 is at least related to lines 08-09 of code; it can also be understood as being related to line 04, or even to line 02. The other tests shown are related to lines of code in a similar manner.

[0220] In this example listing, line 06 has a bug; it should state "month <= 3" instead of "month < 3". But the tests used are all passed, and the line with the bug is covered by the test set. If the developer fixes the bug, the tests will continue to pass as before, and the line on which the fix was made will continue to be shown as covered in the code coverage report. However, the execution scenario when "month = 3" is still not covered. By applying the teachings presented in this article, such a pull request can be marked by the system, and the user is asked to add test coverage for this scenario.

[0221]

[0222]

[0223] Some embodiments help reduce or avoid adding technical debt that is incurred by adding code without thorough automated testing of the code. This can be particularly beneficial in projects using agile development methodologies, or in continuous integration continuous deployment environments, because bugs are easier to fix, and more likely to be fixed correctly, when the code involved is fresh in the minds of developers and has not yet been replaced by the rapid flow of other code considered in an agile or CICD environment. Similarly, when the code is fresh, tests are more likely to be well coordinated with the code under test.

[0224] Some embodiments use or provide a software testing assurance method that includes receiving 828 a pull request 830. The pull request includes a source code portion 218 that indicates the source code to be changed in the repository. The pull request also includes a test section 232 that indicates automated tests associated with the source code. In the above example, for example, the source code portion may be lines 01-15 and the test section may be lines 16-26. The method also includes finding out 708 whether the test section indicates any changes to automated tests associated with the source code. When the test section indicates changes to automated tests associated with the source code, the method includes accepting 808 the pull request, and when the test section does not indicate any changes to automated tests associated with the source code, the method includes rejecting 844 the pull request.

[0225] Some embodiments operate as follows. The embodiment determines that the submitted code includes updates; this may be implicit, for example, by virtue of the code being part of a pull request. The embodiment finds tests that cover the code, for example, by using one or more tools 234, 238. If the tests covering the code have not changed, the embodiment sets 714 a flag. If at least one test covering the code has changed, the embodiment does not set the flag, at least not yet. After determining that the changed tests are not related to the updated portion of the submitted source code, some embodiments may still set the flag.

[0226] In particular, some embodiments recognize situations where at least one test has changed, but the change is not related to any code in the submitted code, and set a flag even if the test has occurred. For example, if the code portion of interest is portion A, and the test covers portions A, B, and C, and the change in the test is only related to B and C and not A, this can occur. Some embodiments recognize situations where at least one test has changed, but the change is not related to (multiple) updated rows in the submitted code. Therefore, even if the test changes and even if the change is related to a part of the portion "P-current", the embodiment sets a flag. For example, if the portion "P-current" has rows 1-10, the update is in row 1 of the portion "P-current", the test covers rows 1-10, and the change in the test is only related to row 3 but not row 1, this can occur.

[0227] Some additional combinations and variations

[0228] Any of these combinations of code, data structures, logic, components, communications, and / or their functional equivalents may also be combined with any of the above systems and variations thereof. A process may include any of the steps described herein in any subset or combination or sequence that is operable. Each variation may occur alone or in combination with any one or more other variations. Each variation may occur with any process, and each process may be combined with any one or more other processes. Each process or combination of processes (including variations) may be combined with any of the above configured storage media combinations and variations.

[0229] in conclusion

[0230] In short, benefiting from the teachings provided herein, embodiments can be used to detect possible vulnerabilities in automated testing of software under development. Complete line coverage by test 232 does not necessarily indicate actual coverage of execution scenarios, for example, even when all source code statements are nominally covered 308, condition coverage 320, decision coverage 306, and other types of execution scenario coverage 302 may also be lacking. When source code changes 222 are submitted and the corresponding test set remains unchanged, the user 104 is notified 836 that sufficient testing 804 is not guaranteed by the current test set. The test guarantee code 204 in the development tool chain 208 can indicate 714 items 228 (such as pull requests 830, 216, test sets 232, or source code 220 submitted 216) to indicate 716 the lack of test assurance. In some cases, the assurance enhancement tools 122, 208, 210, 214, 202 may require that new or different tests be provided with the updated source code, for example, as a prerequisite for source code to be accepted 808 for inclusion 810 in the repository 902 or build 816, or even as a prerequisite for peer review 814. That is, inconsistencies in source code disposition and corresponding test sets are flagged and brought to the attention of developers, testers, and other users to either reduce the risk of test vulnerabilities by obtaining updated tests, or to manage the risk by obtaining informed consent before source code updates proceed in the workflow.

[0231] Although specific embodiments are explicitly shown and described herein as processes, configured storage media, or systems, it should be understood that a discussion of one type of embodiment also generally extends to other types of embodiments. Figure 7 and Figure 8The description of the processes performed also helps describe the configured storage medium, and helps describe the technical effects and operations of systems and manufactures similar to those discussed in conjunction with the other figures. It is not intended that limitations from one embodiment must be read into another embodiment. In particular, the processes are not necessarily limited to the data structures and arrangements presented when discussing systems or manufactures such as configured memory.

[0232] Those skilled in the art will appreciate that implementation details may be associated with specific code, such as specific APIs, specific fields, specific types of components, and specific sample programs, and therefore need not appear in every embodiment. Those skilled in the art will also appreciate that program identifiers and some other terms used when discussing details are implementation-specific, and therefore need not be associated with every embodiment. Nevertheless, although their presence is not necessarily required here, such details may help some readers by providing context, and / or may illustrate some of the many possible implementations of the technology discussed herein.

[0233] References herein to embodiments having some feature X and references elsewhere herein to embodiments having some feature Y do not exclude embodiments having both feature X and feature Y, unless such exclusion is expressly stated herein. All possible negative claim limitations are within the scope of the present disclosure, since any feature specified as part of an embodiment may be expressly deleted from other embodiments even if no specific exclusion is given in any example herein. The term "embodiment" is used herein merely as a more convenient form of "process, system, article, configured computer-readable storage medium, and / or other example of the teachings herein applied in a manner consistent with applicable law." Thus, a given "embodiment" may include any combination of the features disclosed herein, as long as the embodiment is consistent with at least one claim.

[0234] In each embodiment, not every item shown in the figure needs to exist. On the contrary, the embodiment may include (multiple) items that are not explicitly shown in the figure. Although some possibilities are shown in the text and the accompanying drawings by specific examples here, the embodiment may be separated from these examples. For example, the specific technical effects or technical features of the examples may be omitted, renamed, differently aggregated, repeated, differently instantiated in hardware and / or software, or a mixture of effects or features that appear in two or more examples. In some embodiments, the functionality shown at one location may also be provided at different locations; for example, those skilled in the art recognize that functional modules may be defined in various ways in a given implementation without ignoring the required technical effects from a collection of interactive modules that are considered as a whole. Due to space limitations or for convenience, different steps may be shown together in a single frame in the accompanying drawings, but are still executable individually, for example, in a given performance of the method, a step may be performed without another step.

[0235] Reference is made to the drawings throughout the text by reference numerals. Any apparent inconsistency in the wording associated with a given reference numeral in the drawings or text should be understood to simply expand the scope to which that numeral refers. Even when the same reference numeral is used, different instances of a given reference numeral may refer to different embodiments. Similarly, a given reference numeral may refer to a different embodiment.

[0236] can be used to refer to verbs, nouns, and / or corresponding instances of each, e.g., processor 110

[0237] The instructions may be processed 110 by executing the instructions.

[0238] As used herein, terms such as "a", "an" and "the" include one or more of the indicated items or steps. In particular, in the claims, reference to an item generally means that there is at least one such item, and reference to a step indicates at least one instance of performing the step.

[0239] Headings are for convenience only; information on a given topic may be found outside the section whose heading indicates that topic.

[0240] All claims and the abstract as filed are part of the specification.

[0241] Although exemplary embodiments have been shown in the drawings and described above, it is clear to those skilled in the art that various modifications may be made without departing from the principles and concepts set forth in the claims, and such modifications do not need to encompass the entire abstract concept. Although the subject matter is described in a language specific to structural features and / or program actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific technical features or actions described in the claims. Each component or aspect or technical effect identified in a given definition or example does not necessarily exist or is utilized in every embodiment. Instead, the specific features, actions, and effects described are disclosed as examples to be considered when implementing the claims.

[0242] To the maximum extent permitted by law, all changes that come within the general abstract concept and within the meaning and range of equivalency of the claims are intended to be embraced within their scope.

Claims

1. A computing system operable to support software test assurance, the computing system comprising: processor; a memory in operable communication with the processor; as well as Assurance code, after being executed by the processor, the assurance code performs a software test assurance method, the software test assurance method comprising: (a) receiving a code update submission "S-current", the submission "S-current" including a portion "P-current", the portion "P-current" including an update to a source code, (b) determining that the portion "P-current" is at least partially covered by a test set "TS-current" of a test suite, (c) finding out whether the test set "TS-current" is different from a test set "TS-previous", the test set "TS- "TS-Previous" at least partially covers a part of the code update submission "S-Previous" "P-Previous", wherein "S-Previous" is older than "S-Current", and wherein "P-Current" updates "P-Previous", and (d) when the ascertainment ascertains that "TS-Current" is not different from "TS-Previous", marking at least one item based on the "TS-Current" test set to indicate the lack of test guarantee for the "P-Current" update, wherein the marking sets a mark of at least one of the following: "S-Current", "P-Current", or "TS-Current".

2. The system of claim 1, wherein the system comprises an integrated development environment, and the assurance code resides at least partially in the integrated development environment.

3. The system of claim 1, wherein the system comprises an extensible software development tool and the assurance code resides at least in part in an extension to the extensible software development tool.

4. The system of claim 1, wherein the test set TS-current resides in the system and comprises assertions of values ​​that will be produced by execution of at least a portion of the portion P-current.

5. The system of claim 1 , wherein the system is configured to require that any flags set by the assurance code be resolved before the portion “P-current” is accepted, and wherein the portion being accepted means at least one of: the portion being accepted for inclusion in a repository, the portion being accepted for peer review, or the portion being accepted for inclusion in a build.

6. The system of claim 1, wherein the system further comprises at least one of the following: a coverage report, the coverage report being generated by a coverage reporting tool and indicating test coverage of at least a portion of the source code by the test set “TS-current”; or A coverage reporting tool, the coverage reporting tool executable to check test coverage of at least a portion of the source code by the test set "TS-current".

7. A software test assurance method implemented using a computing system, the computing system having at least one processor coupled to instructions stored in a memory, the instructions stored in the memory performing the method when executed by the processor, the method comprising: electronically receiving a code update submission "S-current", said code update submission "S-current" comprising a portion "P-current", said portion "P-current" having one or more lines of source code, said portion "P-current" comprising an update to said source code; automatically determining that the portion "P-current" of the source code is at least partially covered by a test set "TS-current" of the test suite; automatically finding out whether the test set "TS-current" is different from a test set "TS-previous", the test set "TS-previous" at least partially covering a part "P-previous" of a code update commit "S-previous", wherein "S-previous" is older than "S-current", and wherein "P-current" updates "P-previous"; and When the ascertainment ascertains that "TS-current" is not different from "TS-previous", at least one item is automatically marked based on the "TS-current" test set to indicate a lack of test assurance for the update contained in "P-current", and otherwise avoiding marking any item on the basis that "TS-current" is not different from "TS-previous" relative to "P-current", wherein the at least one marked item includes at least one of the following: "S-current", "P-current", or "TS-current".

8. The method of claim 7, wherein the ascertaining ascertains that "TS-current" is different from "TS-previous", and the method further comprises: Recognize that there are no differences between TS-Previous and TS-Current with respect to any tests covering P-Current, and then mark the item to indicate the lack of test assurance for the update being included in P-Current.

9. The method of claim 7, wherein the ascertaining ascertains that "TS-current" is different from "TS-previous", and the method further comprises: Recognize that there are no differences between TS-Previous and TS-Current with respect to any tests covering the update in P-Current, and then mark the entry to indicate the lack of test assurance for the update being included in P-Current.

10. The method of claim 7, wherein receiving the code update commit "S-current" comprises receiving a pull request, the pull request comprising a request for the portion "P-current" to be pulled into a repository, the repository containing the source code.

11. The method of claim 7, wherein determining that the portion “P-current” of the source code is at least partially covered by the test set “TS-current” comprises at least one of the following: Read a coverage report generated by a coverage reporting tool; or A coverage reporting tool is invoked to check coverage of at least a portion of the source code by the test set TS-current.

12. The method according to claim 7, wherein finding out whether the test set "TS-current" is different from the test set "TS-previous" comprises at least one of the following: Check the timestamp of the test set "TS-current"; or An automated test management tool is called, which tracks at least the test set "TS-current".

13. The method of claim 7, wherein at least one item is marked, and wherein the method further comprises notifying the user by displaying at least one of: a notification indicating that a line of source code is at least partially covered by a test, but that the test should be reviewed because the test does not necessarily provide complete coverage; Notifications listing at least two of the following types of code coverage: function coverage, statement coverage, branch coverage, condition coverage, decision coverage, path coverage, entry / exit coverage, loop coverage, finite state machine state coverage, or data flow coverage.

14. A storage medium configured with code, which, upon execution by one or more processors, performs a software test assurance method, the method comprising: Receiving a code update submission "S-current", wherein the code update submission "S-current" includes a portion "P-current", wherein the portion "P-current" includes updated source code; determining that the portion “P-current” is at least partially covered by a test set “TS-current” of the test suite; Find out whether the test set "TS-current" is different from the test set "TS-previous", the test set "TS-previous" at least partially covering a part "P-previous" of the code update commit "S-previous", wherein "S-previous" is older than "S-current", and wherein "P-current" updates "P-previous"; and When the ascertainment ascertains that "TS-current" is not different from "TS-previous", at least one item is marked based on the "TS-current" test set to indicate a lack of test assurance for "P-current", and otherwise avoiding marking any item on the basis that "TS-current" is not different from "TS-previous" relative to "P-current", wherein the at least one marked item includes at least one of the following: "S-current", "P-current", or "TS-current".

15. The storage medium of claim 14, wherein receiving the code update commit "S-current" comprises receiving a pull request, the pull request comprising a request for the portion "P-current" to be pulled into a repository.

16. The storage medium of claim 14, wherein the ascertaining ascertains that "TS-current" is different from "TS-previous", and the method further comprises: Recognize that there are no differences between TS-Previous and TS-Current with respect to any tests covering any portion of P-Current, and then mark the item to indicate the lack of test assurance for the update being included in P-Current.

17. The storage medium of claim 14, wherein the ascertaining ascertains that "TS-current" is different from "TS-previous", and the method further comprises: Recognize that there are no differences between TS-Previous and TS-Current with respect to any tests covering the update in P-Current, and then mark the entry to indicate the lack of test assurance for the update being included in P-Current.

18. The storage medium of claim 14, wherein determining that the portion P-current of source code is at least partially covered by the test set TS-current comprises: Reads data generated by the coverage reporting tool.

19. The storage medium of claim 14, wherein finding out whether the test set TS-current is different from the test set TS-previous comprises at least one of the following: Check the timestamp of the test set "TS-current"; or Communicates with automated test management tools.

20. The storage medium of claim 14, wherein at least one item is marked, and wherein the method further comprises notifying a user by displaying a notification indicating that at least a portion of the "P-current" source code is at least partially covered by tests, but that because the tests do not necessarily provide complete coverage, the tests should be reviewed.

Images