Methods, systems, and computer program products for automatically selecting tests for regression testing of software systems using machine learning.

Abstract

What is provided is a computer-implemented method, system, and computer program product for automatically selecting tests for regression testing of a software system using machine learning. This includes generating a test map that includes at least one of a plurality of tests corresponding to source files. The plurality of tests and at least one source file are associated with a software repository. Furthermore, the method involves determining a defect score for the at least one test based on historical test data; receiving a component criticality score and defect definition corresponding to the source file; generating a key value corresponding to the at least one test based on the defect score, component criticality score, and defect definition; determining a subset of the plurality of tests based on the key value corresponding to the at least one test; and executing the test subset in conjunction with the software repository.

Description

Technical Field

[0001] The subject matter of this disclosure generally relates to methods, systems, and products for regression testing of software systems, and in some specific embodiments or aspects, to methods, systems, and computer program products for automatically selecting tests for regression testing of software systems. Background Technology

[0002] Software systems typically require thorough verification checks and testing to confirm that they functionally meet project requirements. Testing also helps minimize vulnerabilities and other unwanted features, enabling high-quality software system delivery. Due to the incremental nature of system changes throughout the development lifecycle, software systems often require more frequent quality checks and testing. For software systems, quality checks and testing include regression testing, which executes all software tests to detect vulnerabilities and defects that may be introduced through incremental changes (e.g., code changes, code check-ins, deployments, integrations).

[0003] Typically, regression testing is performed after each incremental change to a software system. For example, after software developers integrate code changes to a feature (e.g., new features, modifications, implementations), the entire regression test suite is executed to catch any defects introduced by that specific code change. Executing the entire regression test suite can be time-consuming, and the benefit of executing the entire suite may be minimal when integrating small changes into the system. Specific tests can be manually selected after integrating code changes, but this can be very time-consuming and resource-intensive, and may result in tests being missed without being executed.

[0004] To maintain system quality, regression testing should be executed systematically, ensuring that tests are relevant to code changes introduced into the system, but without being excessively time-consuming or resource-intensive. Code changes to the system may be frequent and can be integrated by multiple developers simultaneously or at different times. Executing tests in the regression test suite that are unrelated to each code change can be time-consuming and resource-intensive. Summary of the Invention

[0005] Therefore, improved methods, systems, and products for regression testing of software systems are provided.

[0006] According to some non-limiting embodiments or aspects, a computer-implemented method is provided, comprising: generating a test map using at least one processor, the test map including at least one test from a plurality of tests corresponding to at least one source file, wherein the plurality of tests and the at least one source file are associated with a software repository; determining a defect score for the at least one test based on historical test data of the at least one test using the at least one processor and at least one machine learning model; receiving a component criticality score and a defect definition corresponding to the at least one source file using the at least one processor; generating a key value corresponding to the at least one test based on the defect score, the component criticality score, and the defect definition using the at least one processor and the at least one machine learning model; determining a subset of tests from the plurality of tests based on the key value corresponding to the at least one test using the at least one processor and the at least one machine learning model; and executing the subset of tests using the at least one processor together with the software repository.

[0007] In some non-limiting embodiments or aspects, the computer-implemented method further includes: generating test results using the at least one processor based on the execution of the test subset; and inputting the test results into the at least one machine learning model to further train the at least one machine learning model.

[0008] In some non-limiting embodiments or aspects, the computer-implemented method further includes: generating the test map by: detecting changes to at least one source file based on changes to at least one source file, wherein the changes include differences between a first version of the at least one source file and a second version of the at least one source file; selecting at least one test corresponding to the at least one source file based on a predefined source-to-test mapping; compiling a list of source files and a list of tests, wherein the list of source files includes source filenames of the at least one source file, and the list of tests includes test filenames of the at least one test; and generating the test map from the list of source files and the list of tests by mapping the source filenames to their corresponding test filenames based on the predefined source-to-test mapping.

[0009] In some non-limiting embodiments or aspects, the computer-implemented method further includes: the at least one source file is a new source file, and the change includes adding a new source file to the software repository.

[0010] In some non-limiting embodiments or aspects, the computer-implemented method further includes: determining the test subset by: generating a recommended test subset based on the key value and a predefined test category; generating a mapping file based on the recommended test subset and the at least one source file; storing the mapping file in a database; and formatting the recommended test subset into an executable format.

[0011] In some non-limiting embodiments or aspects, the computer-implemented method further includes: the at least one machine learning model using supervised learning and prediction to determine the test subset.

[0012] In some non-limiting embodiments or aspects, the computer-implemented method further includes: using a mapping file stored in the database as input to the at least one machine learning model for further training.

[0013] According to some non-limiting embodiments or aspects, a system is provided, comprising: a test selection system; at least one machine learning model; a software repository; and at least one processor, wherein the at least one processor is programmed or configured to: generate a test map including at least one of a plurality of tests corresponding to at least one source file, wherein the plurality of tests and the at least one source file are associated with the software repository; determine a defect score for the at least one test based on historical test data of the at least one test; receive a component criticality score and a defect definition corresponding to the at least one source file; generate a key value corresponding to the at least one test based on the defect score, the component criticality score, and the defect definition; determine a subset of the plurality of tests based on the key value corresponding to the at least one test; and execute the subset of tests together with the software repository.

[0014] In some non-limiting embodiments or aspects, the system further includes: the at least one processor being further programmed or configured to: generate test results based on the execution of the test subset; and input the test results into the at least one machine learning model to further train the at least one machine learning model.

[0015] In some non-limiting embodiments or aspects, the system further includes: generating the test map, wherein the at least one processor is programmed or configured to: detect the at least one source file based on changes to the at least one source file, wherein the changes include differences between a first version and a second version of the at least one source file; select at least one test corresponding to the at least one source file based on a predefined source-to-test mapping; compile a list of source files and a list of tests, wherein the list of source files includes source filenames of the at least one source file, and the list of tests includes test filenames of the at least one test; and generate the test map from the list of source files and the list of tests by mapping the source filenames to their corresponding test filenames based on the predefined source-to-test mapping.

[0016] In some non-limiting embodiments or aspects, the system further includes: the at least one source file is a new source file, and the change includes a new source file added to the software repository.

[0017] In some non-limiting embodiments or aspects, the system further includes: when the test subset is determined, the at least one processor is programmed or configured to: generate a recommended test subset based on the key value and a predefined test category; generate a mapping file based on the recommended test subset and the at least one source file; store the mapping file in a database; and format the recommended test subset into an executable format.

[0018] In some non-limiting embodiments or aspects, the system further includes: the at least one machine learning model using supervised learning and prediction to determine the test subset.

[0019] In some non-limiting embodiments or aspects, the system further includes: a mapping file stored in the database used as input to the at least one machine learning model for further training.

[0020] According to some non-limiting embodiments or aspects, a computer program product is provided, the computer program product comprising at least one non-transitory computer-readable medium, the at least one non-transitory computer-readable medium comprising one or more instructions, the one or more instructions, when executed by at least one processor, causing the at least one processor to: generate a test map, the test map comprising at least one of a plurality of tests corresponding to at least one source file, wherein the plurality of tests and the at least one source file are associated with a software repository; determine a defect score for the at least one test based on historical test data of the at least one test; receive a component criticality score and a defect definition corresponding to the at least one source file; generate a key value corresponding to the at least one test based on the defect score, the component criticality score, and the defect definition; determine a subset of tests of the plurality of tests based on the key value corresponding to the at least one test; and execute the subset of tests together with the software repository.

[0021] In some non-limiting embodiments or aspects, the computer program product further includes: the one or more instructions further causing the at least one processor to: generate test results based on the execution of the test subset; and input the test results into the at least one machine learning model to further train the at least one machine learning model.

[0022] In some non-limiting embodiments or aspects, the computer program product further includes: when generating the test map, the one or more instructions further cause the at least one processor to: detect the at least one source file based on changes to the at least one source file, wherein the changes include differences between a first version of the at least one source file and a second version of the at least one source file; select at least one test corresponding to the at least one source file based on a predefined source file-to-test mapping; compile a list of source files and a list of tests, wherein the list of source files includes source filenames of the at least one source file, and the list of tests includes test filenames of the at least one test; and generate the test map from the list of source files and the list of tests by mapping the source filenames to their corresponding test filenames based on the predefined source file-to-test mapping.

[0023] In some non-limiting embodiments or aspects, the computer program product further includes: the at least one source file is a new source file, and the change includes a new source file added to the software repository.

[0024] In some non-limiting embodiments or aspects, the computer program product further includes: when the test subset is determined, the one or more instructions further cause the at least one processor to: generate a recommended test subset based on the key value and a predefined test category; generate a mapping file based on the recommended test subset and the at least one source file; store the mapping file in a database; and format the recommended test subset into an executable format.

[0025] In some non-limiting embodiments or aspects, the computer program product further includes: a mapping file stored in the database used as input to the at least one machine learning model for further training.

[0026] Further embodiments or aspects are set forth in the following numbered clauses:

[0027] Clause 1: A computer-implemented method comprising: generating a test map using at least one processor, the test map including at least one test from a plurality of tests corresponding to at least one source file, wherein the plurality of tests and the at least one source file are associated with a software repository; determining a defect score for the at least one test based on historical test data of the at least one test using the at least one processor and at least one machine learning model; receiving a component criticality score and a defect definition corresponding to the at least one source file using the at least one processor; generating a key value corresponding to the at least one test based on the defect score, the component criticality score, and the defect definition using the at least one processor and the at least one machine learning model; determining a subset of tests from the plurality of tests based on the key value corresponding to the at least one test using the at least one processor and the at least one machine learning model; and executing the subset of tests using the at least one processor in conjunction with the software repository.

[0028] Clause 2: The computer-implemented method according to Clause 1 further includes: generating test results using the at least one processor based on the execution of the test subset; and inputting the test results into the at least one machine learning model to further train the at least one machine learning model.

[0029] Clause 3: A computer-implemented method according to Clause 1 or 2, wherein generating the test map comprises: detecting at least one source file based on changes to at least one source file, wherein the changes include differences between a first version of the at least one source file and a second version of the at least one source file; selecting at least one test corresponding to the at least one source file based on a predefined source-to-test mapping; compiling a list of source files and a list of tests, wherein the list of source files includes source filenames of the at least one source file, and the list of tests includes test filenames of the at least one test; and generating the test map from the list of source files and the list of tests by mapping the source filenames to their corresponding test filenames based on the predefined source-to-test mapping.

[0030] Clause 4: A computer-implemented method according to any one of Clauses 1-3, wherein the at least one source file is a new source file, and the change includes a new source file added to the software repository.

[0031] Clause 5: A computer-implemented method according to any one of Clauses 1-4, wherein determining the test subset comprises: generating a recommended test subset based on the key value and a predefined test category; generating a mapping file based on the recommended test subset and the at least one source file; storing the mapping file in a database; and formatting the recommended test subset into an executable format.

[0032] Clause 6: A computer-implemented method according to any one of Clauses 1-5, wherein the at least one machine learning model uses supervised learning and prediction to determine the test subset.

[0033] Clause 7: A computer-implemented method according to any one of Clauses 1-6, wherein a mapping file stored in the database is used as input to the at least one machine learning model for further training.

[0034] Clause 8: A system comprising: a test selection system; at least one machine learning model; a software repository; and at least one processor, wherein the at least one processor is programmed or configured to: generate a test map including at least one of a plurality of tests corresponding to at least one source file, wherein the plurality of tests and the at least one source file are associated with the software repository; determine a defect score for the at least one test based on historical test data of the at least one test; receive a component criticality score and a defect definition corresponding to the at least one source file; generate a key value corresponding to the at least one test based on the defect score, the component criticality score, and the defect definition; determine a subset of tests of the plurality of tests based on the key value corresponding to the at least one test; and execute the subset of tests together with the software repository.

[0035] Clause 9: The system according to Clause 8, wherein the at least one processor is further programmed or configured to: generate test results based on the execution of the test subset; and input the test results into the at least one machine learning model to further train the at least one machine learning model.

[0036] Clause 10: The system according to Clause 8 or 9, wherein, when generating the test map, the at least one processor is programmed or configured to: detect the at least one source file based on changes to the at least one source file, wherein the changes include differences between a first version and a second version of the at least one source file; select at least one test corresponding to the at least one source file based on a predefined source-to-test mapping; compile a list of source files and a list of tests, wherein the list of source files includes source filenames of the at least one source file, and the list of tests includes test filenames of the at least one test; and generate the test map from the list of source files and the list of tests by mapping the source filenames to their corresponding test filenames based on the predefined source-to-test mapping.

[0037] Clause 11: A system pursuant to any one of Clauses 8-10, wherein the at least one source file is a new source file, and the change includes a new source file added to the software repository.

[0038] Clause 12: A system according to any one of Clauses 8-11, wherein, when the test subset is determined, the at least one processor is programmed or configured to: generate a recommended test subset based on the key value and a predefined test category; generate a mapping file based on the recommended test subset and the at least one source file; store the mapping file in a database; and format the recommended test subset into an executable format.

[0039] Clause 13: A system according to any one of Clauses 8-12, wherein the at least one machine learning model uses supervised learning and prediction to determine the test subset.

[0040] Clause 14: A system according to any one of Clauses 8-13, wherein a mapping file stored in the database is used as input to the at least one machine learning model for further training.

[0041] Clause 15: A computer program product comprising at least one non-transitory computer-readable medium, the at least one non-transitory computer-readable medium comprising one or more instructions, the one or more instructions, when executed by at least one processor, causing the at least one processor to: generate a test map, the test map comprising at least one of a plurality of tests corresponding to at least one source file, wherein the plurality of tests and the at least one source file are associated with a software repository; determine a defect score for the at least one test based on historical test data of the at least one test; receive a component criticality score and a defect definition corresponding to the at least one source file; generate a key value corresponding to the at least one test based on the defect score, the component criticality score, and the defect definition; determine a subset of tests based on the key value corresponding to the at least one test; and execute the subset of tests together with the software repository.

[0042] Clause 16: A computer program product according to Clause 15, wherein the one or more instructions further cause the at least one processor to: generate test results based on the execution of the test subset; and input the test results into at least one machine learning model to further train the at least one machine learning model.

[0043] Clause 17: A computer program product according to Clause 15 or 16, wherein, when generating the test map, the one or more instructions further cause the at least one processor to: detect the at least one source file based on changes to the at least one source file, wherein the changes include differences between a first version of the at least one source file and a second version of the at least one source file; select at least one test corresponding to the at least one source file based on a predefined source-to-test mapping; compile a list of source files and a list of tests, wherein the list of source files includes source file names of the at least one source file, and the list of tests includes test file names of the at least one test; and generate the test map from the list of source files and the list of tests by mapping the source file names to their corresponding test file names based on the predefined source-to-test mapping.

[0044] Clause 18: A computer program product pursuant to any one of Clauses 15-17, wherein the at least one source file is a new source file, and the change includes a new source file added to the software repository.

[0045] Clause 19: A computer program product according to any one of Clauses 15-18, wherein, when the test subset is determined, the one or more instructions further cause the at least one processor to: generate a recommended test subset based on the key value and a predefined test category; generate a mapping file based on the recommended test subset and the at least one source file; store the mapping file in a database; and format the recommended test subset into an executable format.

[0046] Clause 20: A computer program product according to any one of Clauses 15-19, wherein a mapping file stored in the database is used as input to at least one machine learning model for further training.

[0047] These and other features and characteristics of the subject matter, as well as the operational methods and functions of the associated structural elements and combinations of parts, and the economics of manufacture, will become more apparent upon consideration of the following description and appended claims with reference to the accompanying drawings, all of which form part of this specification, wherein like reference numerals denote corresponding parts in the figures. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to be a definition of limitation on the disclosed subject matter. Unless the context clearly specifies otherwise, the singular forms “a” and “described” as used in this specification and claims include plural indicators. Attached Figure Description

[0048] Additional advantages and details of the disclosed subject matter will be explained in more detail below with reference to exemplary embodiments or aspects illustrated in the accompanying drawings, in which:

[0049] Figure 1 This is a schematic diagram of a system for automatically selecting tests for regression testing of a software system using machine learning, based on some non-limiting embodiments or aspects.

[0050] Figure 2 This is a schematic diagram of an exemplary test selection system for automatically selecting tests for regression testing of a software system using machine learning, according to some non-limiting embodiments or aspects.

[0051] Figure 3 It is a flowchart of a non-limiting embodiment or aspect of a process for automatically selecting tests for regression testing of a software system using machine learning, based on the principles of the currently disclosed subject matter;

[0052] Figure 4 yes Figure 1A diagram of a non-limiting embodiment or aspect of a component of one or more devices; and

[0053] Figure 5 This is a schematic diagram of components of a test selection system for automatically selecting tests for regression testing of a software system using machine learning, according to some non-limiting embodiments or aspects. Detailed Implementation

[0054] For descriptive purposes, the terms “end,” “upper,” “lower,” “right,” “left,” “vertical,” “horizontal,” “top,” “bottom,” “lateral,” “longitudinal,” and their derivatives are intended to refer to the orientation of the disclosed subject matter as shown in the accompanying drawings. However, it should be understood that the disclosed subject matter may take various alternative variations and sequences of steps, except where explicitly specified otherwise. It should also be understood that the specific apparatus and processes shown in the drawings and described in the following specification are merely exemplary embodiments or aspects of the disclosed subject matter. Therefore, unless otherwise indicated, specific dimensions and other physical characteristics associated with the embodiments or aspects disclosed herein should not be considered limiting.

[0055] The terms "aspects," "components," "elements," "elements," "structures," "actions," "steps," "functions," and "instructions" used herein should not be construed as critical or essential unless explicitly stated otherwise. Furthermore, as used herein, the article "a" is intended to include one or more items and is interchangeable with "one or more" and "at least one." Additionally, as used herein, the term "set" is intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and is interchangeable with "one or more" or "at least one." Where only one item is desired, the term "a" or similar language is used. Furthermore, as used herein, the terms "having" and similar expressions are intended to be open-ended terms. Additionally, unless explicitly stated otherwise, the phrase "based on" is intended to mean "at least partially based on."

[0056] As used herein, the terms "communication" and "transmission" can refer to the receiving, accepting, sending, transmitting, or providing of information (e.g., data, signals, messages, instructions, commands, etc.). For one unit (e.g., a device, system, component of a device or system, or a combination thereof) to communicate with another unit means that the first unit is able to receive information directly or indirectly from and / or send information to the other unit. This can refer to a direct or indirect connection that is inherently wired and / or wireless (e.g., a direct communication connection, an indirect communication connection, etc.). Furthermore, although the transmitted information may be modified, processed, relayed, and / or routed between the first and second units, the two units can also communicate with each other. For example, the first unit can communicate with the second unit even if it passively receives information and does not actively send information to the second unit. As another example, the first unit can communicate with the second unit if at least one intermediate unit (e.g., a third unit located between the first and second units) processes information received from the first unit and transmits the processed information to the second unit. In some non-limiting embodiments or aspects, a message may refer to a network packet that includes data (e.g., a data packet, etc.). It should be understood that many other arrangements are possible.

[0057] As used herein, the term "computing device" can refer to one or more electronic devices configured to process data. In some examples, a computing device may include the necessary components for receiving, processing, and outputting data, such as a processor, display, memory, input device, network interface, etc. A computing device can be a mobile device. As examples, a mobile device may include a cellular phone (e.g., a smartphone or standard cellular phone), a portable computer, a wearable device (e.g., a watch, glasses, lenses, clothing, etc.), a personal digital assistant (PDA), and / or other similar devices. A computing device can also be a desktop computer or other forms of non-mobile computer.

[0058] As used herein, the term "server" may refer to one or more computing devices (e.g., processors, storage devices, similar computer components, etc.) that communicate with client devices and / or other computing devices via a network (e.g., a public network, the Internet, a private network, etc.) and, in some examples, facilitate communication between other servers and / or client devices. It should be understood that various other arrangements are possible.

[0059] As used herein, the term "system" may refer to one or more computing devices or a combination of computing devices (e.g., processor, server, client device, software application, components of such computing devices, etc.). References to "device," "server," "processor," etc., as used herein, may refer to a previously stated device, server, or processor, a different server or processor, and / or a combination of servers and / or processors, stated as performing a prior step or function. For example, as used in the specification and claims, a first server or first processor stated as performing a first step or a first function may refer to the same or different server or the same or different processor stated as performing a second step or a second function.

[0060] Non-limiting embodiments or aspects of the disclosed subject matter relate to automated test selection, including but not limited to systems, methods, and computer program products that use machine learning to automatically select tests for regression testing of software systems. For example, non-limiting embodiments or aspects of the disclosed subject matter provide: generating a test map including tests (e.g., test files, test cases, test objects, test methods, etc.), said tests corresponding to source files (e.g., source code, source code files, project files, etc.), wherein the tests and source files are included in a software repository (e.g., code repository, project, code project, software project, etc.); determining a defect score for each test based on historical test data, said historical test data including test failures (e.g., previous test failures, recorded test failures, stored test failures, etc.) and test executions (e.g., previous test executions, recorded test executions, stored test executions, etc.); receiving criticality scores and defect definitions for components (e.g., software components, system components, etc.) corresponding to the source files; generating a critical value corresponding to at least one entry of the test map based on the defect scores, component criticality scores, and defect definitions; determining a subset of tests based on the critical values ​​corresponding to at least one entry of the test map; and executing (e.g., running, etc.) the subset of tests together with the software repository. These embodiments provide technologies and systems that use machine learning to automatically select tests from a system test suite used for regression testing to achieve maximum code coverage, while executing only the necessary tests in the software repository based on detected code changes (e.g., code check-ins, deployments, file changes, push changes, code integrations, etc.). These embodiments can improve continuous integration activities in software development projects and require fewer computational resources to test the software repository when integrating code changes. These embodiments can reduce the amount of time required to perform regression tests within the software repository. These embodiments can allow training machine learning models to continuously improve test selection based on previous test selections and executions. These embodiments can select tests corresponding to the detected code changes to be integrated and allow selecting the most relevant tests for each detected code change. These embodiments can enable product quality improvements in continuous integration pipelines where changes to the software repository are frequently integrated.

[0061] For illustrative purposes, while the subject matter disclosed herein describes methods, systems, and computer program products for automated test selection (e.g., for regression testing of software systems), those skilled in the art will recognize that the disclosed subject matter is not limited to the illustrative embodiments or aspects. For example, the methods, systems, and computer program products described herein can be used with a wide variety of settings, such as any software system and / or integrated software hardware system suitable for using such automated test selection (e.g., any application using tests to test a system).

[0062] Now for reference Figure 1 , Figure 1 This is a schematic diagram of a system 100 for automatically selecting tests for regression testing of a software system, according to some non-limiting embodiments or aspects. System 100 may include a test selection system 102, a machine learning model 104, and a software repository 106. Test selection system 102 may include one or more computing devices that execute one or more software applications configured for continuous integration in software development. Continuous integration may include applying code changes, detecting code changes using a version control system, integrating code changes into software repository 106, building software repository 106 into an executable software project, and performing tests with the executable project, etc. Test selection system 102 may communicate with machine learning model 104 and software repository 106. In some non-limiting embodiments or aspects, test selection system 102 may include machine learning model 104 and software repository 106 and may execute on the same computing device. In other non-limiting embodiments or aspects, each of test selection system 102, machine learning model 104, and software repository 106 may reside on and execute on one or more separate computing devices and may communicate via a communication network. In some non-limiting embodiments or aspects, the test selection system 102 may include one or more storage components residing on one or more computing devices, or may communicate with said one or more storage components.

[0063] Machine learning model 104 may include an artificial intelligence model (e.g., an artificial neural network, a convolutional neural network, a deep learning model, etc.). Machine learning model 104 may communicate with test selection system 102 and software storage library 106. Machine learning model 104 may communicate with one or more storage components 408 residing on one or more computing devices.

[0064] The software repository 106 may include files corresponding to software projects, such as source files, test files, etc. The software repository 106 may reside on one or more computing devices and may be able to be transferred from one computing device to another via a communication network. The software repository 106 may communicate with one or more databases residing on one or more computing devices.

[0065] Now for reference Figure 2 , Figure 2This is a test selection system used to automatically select tests for regression testing of software systems using machine learning. The test selection system 200 may include a software repository 202, tests 204, source files 206, a test list 208, a source file list 210, test mappings 212, a machine learning model 214, a database 216, mapping files 218, and test results 220.

[0066] like Figure 2 As shown, software repository 202 may include tests 204 and source files 206. Software repository 202 may be the same as or similar to software repository 106. In some non-limiting embodiments or aspects, software repository 202 may operate in conjunction with a version control system. For example, software repository 202 may be combined with... or The software repository 202 can be used in conjunction with other version control systems to enable the detection of code changes within the software repository 202. In some non-limiting embodiments or aspects, the software repository 202 can be combined with an automation server (e.g., Software repository 202 may be copied (e.g., cloned) for testing and development purposes. Software repository 202 may exist as multiple copies residing simultaneously on different computing devices. Software repository 202 may exist as multiple different versions (e.g., including different code changes between versions), and different versions may reside on different computing devices (e.g., device 400).

[0067] Test 204 may include software tests (e.g., test files, test cases, test methods, automation scripts, etc.). Test 204 may include newly developed tests, regression tests, unit tests, component tests, integration tests, etc. In some non-limiting embodiments or aspects, Test 204 may be categorized into multiple types of tests. Test 204 may be grouped into packages, files, modules, or similar grouping strategies. Test 204 may be written in any suitable software language, as long as Test 204 can be formatted into an executable format and executed by processor 404. When Test 204 is changed or edited, the version control system can detect the code changes to Test 204. Code changes to Test 204 may include deleting tests, editing tests, renaming tests, or creating new tests.

[0068] Continue to refer to Figure 2Source file 206 may include newly developed source files, definition files, implementation files, class files, configuration files, documentation files, user interface files, etc. Source file 206 may be grouped into packages, components, modules, or similar grouping strategies. In some non-limiting embodiments or aspects, a component may be encompassed by a single source file 206. Source file 206 may be written in any suitable software language, as long as it can be interpreted by a computing device and formatted into an executable format for execution by processor 404. When source file 206 is changed or edited, the version control system can detect the code changes to source file 206. Code changes to source file 206 may include deleting a source file, editing a source file, renaming a source file, or creating a new source file.

[0069] Test list 208 may include a list of test names, test file names, or other suitable identifiers (e.g., hashes) to identify tests. Test list 208 may include identifiers for tests 204 corresponding to source file 206 based on a predefined mapping from test 204 to source file 206. Test list 208 may be generated by processor 404.

[0070] Continue to refer to Figure 2 The source file list 210 may include a list of source file names or other suitable identifiers (e.g., hashes) to identify source files. The source file list 210 may include identifiers for source files 206 corresponding to code changes detected by the version control system. The source file list 210 may be generated by the processor 404.

[0071] Test mapping 212 may include a mapping from test 204 to source file 206. For example, test mapping 212 may include a data structure that includes the mapping between test 204 and source file 206 (e.g., a correspondence of records). The mapping from test 204 to source file 206 may include correspondences, such as one-to-one, one-to-many, or many-to-many correspondences. The correspondence of records can be structured in any conceivable manner, as long as test mapping 212 can be interpreted by a computing device and / or machine learning model 214. Test mapping 212 may be generated from test list 208 and source file list 210. Test mapping 212 may be input into machine learning model 214 to train machine learning model 214 and generate mapping file 218. Each test-source file mapping in test mapping 212 may be assigned one or more key values ​​by machine learning model 214. In some non-limiting embodiments or aspects, test mapping 212 may include tests 204 from multiple categories of test list 208 mapped to tests 204 and source file 206. In some non-limiting embodiments or aspects, the test map 212 may be stored for future training of the machine learning model 214.

[0072] Continue to refer to Figure 2 Machine learning model 214 may include artificial intelligence models (e.g., artificial neural networks, convolutional neural networks, deep learning models, etc.). In some non-limiting embodiments or aspects, machine learning model 214 may be the same as or similar to machine learning model 104. In some non-limiting embodiments or aspects, test selection system 200 may include one or more machine learning models 214. Machine learning model 214 may generate test mappings 212 from test list 208 and source file list 210. Machine learning model 214 may receive test mappings 212 as input to generate mapping file 218 and / or train machine learning model 214. In some non-limiting embodiments or aspects, machine learning model 214 may communicate with database 216.

[0073] In some non-limiting embodiments or aspects, machine learning model 214 may receive information from database 216, such as historical test mapping files, historical mapping files, and historical test results. Machine learning model 214 may receive historical defect scores, historical component criticality scores, and historical defect definitions from database 216. Machine learning model 214 may also receive defect scores, component criticality scores, and defect definitions corresponding to test 204 or source file 206 identified in test mapping 212 from database 216.

[0074] Machine learning model 214 can generate a mapping file 218. Machine learning model 214 can first generate a recommended subset of tests to be included in the mapping file 218. Machine learning model 214 can generate the recommended subset of tests based on key values ​​corresponding to test mapping 212. Machine learning model 214 can generate a key value as output for each test-to-source file entry in test mapping 212. Key values ​​can be generated based on defect scores, component criticality scores, and defect definitions. In some non-limiting embodiments or aspects, machine learning model 214 can receive test results 220 as input to train machine learning model 214.

[0075] Continue to refer to Figure 2Mapping file 218 may contain a subset of tests identified as being executed with test 204 in conjunction with software repository 202. In some non-limiting embodiments or aspects, mapping file 218 may include a data structure comprising mappings (e.g., recorded correspondences) between test 204, source file 206, and / or test categories. The mapping may include one or more entries for the recorded correspondences. For example, mapping file 218 may include a data structure for test 204 with recorded correspondences to source file 206 and / or test categories, wherein recommended tests are included in mapping file 218 with recorded correspondences to one or more test categories and one or more source files 206. The mapping may include correspondences, such as one-to-one, one-to-many, and / or many-to-one correspondences. Recorded correspondences can be structured in any conceivable manner, provided that mapping file 218 can be interpreted by computing devices and / or machine learning model 214. Test 204 listed in mapping file 218 may represent a subset of tests to be executed with the executable software project. In some non-limiting embodiments or aspects, the test subset may or may not be the same as, similar to, and / or include the tests identified in the test map 204. The test subset in the map file 218 may be determined by a recommended test subset generated by machine learning model 214 or another machine learning model based on machine learning model 214. The map file 218 may be generated by machine learning model 214. The map file 218 may be input into another machine learning model and / or software component for the purpose of reading the test subset, obtaining the test subset identified in the map file 218, and formatting each test in the test subset into an executable format. In some non-limiting embodiments or aspects, the map file 218 may be stored in database 216 for future use.

[0076] Test results 220 may include indications of the failure or success of each executed test in the subset of tests identified in mapping file 218. In some non-limiting embodiments or aspects, test results 220 may include additional information with each pass / fail indication, such as execution duration, identification of the covered source file 206, percentage of the covered source file 206, etc. For example, test results 220 may include a table where rows may include test 204, the number of times test 204 failed, an indication of whether test 204 passed or failed during execution, the execution duration of test 204, the percentage of covered code within source file 206, etc. Test results 220 may be structured in any conceivable manner, provided that test results 220 can be interpreted by computing devices and / or machine learning model 214. Test results 220 may include information such as identifiers of code changes (e.g., pull request numbers, dump hashes, etc.), information about affected components or affected source files 206. Test results 220 may include metrics for whether detected code changes were integrated or rejected. For example, test results 220 can determine whether a threshold percentage of a subset of tests is met (e.g., compliant or exceeded) before integrating detected code changes. Test results 220 can be fed into machine learning model 214 for training. Alternatively, test results 220 can be stored in database 216 for later use or training.

[0077] Now for reference Figure 3 , Figure 3 This is a process 300 for automatically selecting tests for system regression testing using machine learning, based on some non-limiting embodiments or aspects. Figure 3 The steps shown are for illustrative purposes only. It should be understood that in non-limiting embodiments or aspects, nearby, fewer, different, and / or different orders of steps may be used. In some non-limiting embodiments or aspects, one or more steps of process 300 may (e.g., wholly, partially, etc.) be performed by one or more systems 100. In some non-limiting embodiments or aspects, one or more steps of the process for automatically testing the selected process may (e.g., wholly, partially, etc.) be performed by another system, another device, another set of systems, or another set of devices that are separate from or include the first computing device.

[0078] like Figure 3 As shown and referenced Figure 1 and 2At step 302, process 300 may include generating test mapping 212. For example, test selection systems 100 and 200 may generate test mapping 212, which includes multiple tests 204 corresponding to source files 206 associated with software repositories 106 and 202. Test mapping 212 may correspond to code changes detected in source files 206 associated with software repositories 106 and 202. In some non-limiting embodiments or aspects, test selection systems 100 and 200 may generate test mapping 212 based on predefined mappings from source files to tests (e.g., test files, test cases, etc.). In some non-limiting embodiments or aspects, the predefined mappings may be one-to-one, one-to-many, or many-to-one mappings. The predefined mappings from source files to tests may be based on component grouping, use case flow, data flow tests of source files, control flow tests of source files, etc. Each source file 206 in test mapping 212 may be associated with a corresponding code change to the source file and / or related source files. The relevant source files may include source files 206 that belong to the common components.

[0079] In some non-limiting embodiments or aspects, test map 212 may be automatically generated in response to the detection of code changes in source file 206. Test selection systems 100, 200 may generate test map 212 by selecting tests 204 corresponding to source file 206 containing code changes based on a predefined mapping from source file to test. In some non-limiting embodiments or aspects, code change detection may occur when code changes are integrated with software repositories 106, 202 (e.g., check-in, deployment, push, etc.). Additionally or alternatively, code changes may be detected after changes are made to the source file but before the source file is integrated with software repositories 106, 202. Regardless of the stage of development in code change detection, when the version control system detects a code change, test map 212, including the mapping from test to source file, may be generated.

[0080] Continue to refer to Figure 3As shown at step 304, process 300 may include determining a defect score for each test 204 based on historical test data (e.g., among a plurality of tests 204 within software repositories 106, 202). For example, test selection systems 100, 200 may determine the defect score for a test 204 corresponding to a source file 206 by indicating the total number of failures of the test 204 during its lifetime when executed with the corresponding source file 206 and / or executable software project during the lifetime of software repositories 106, 202. Additionally or alternatively, test selection systems 100, 200 may determine the defect score for a test 204 by indicating the total number of failures of the test 204 during the lifetime of software repositories 106, 202, regardless of the corresponding source file 206. In some non-limiting embodiments or aspects, the defect score may be determined by assigning a defect score based on a scoring algorithm that includes a predetermined scoring scale. For example, a defect score of 1-3 may be assigned to a test. If the historical test data for a test 204 includes a number of failures less than a threshold number for the test 204, a defect score of 1 may be assigned to the test 204. If the historical test data for test 204 includes a number of failures exceeding a threshold, a defect score of 3 can be assigned to test 204. If the historical test data for test 204 includes a number of failures within a specified range, a defect score of 2 can be assigned to test 204. It should be understood that the scoring algorithm provided herein is an exemplary, non-limiting embodiment provided for illustrative purposes, and the scoring algorithm can be any feasible scoring algorithm used to determine the defect score of test 204. The scoring algorithm can include any such scale, calculation, formula, representation, etc., that measures the defect of test 204.

[0081] In some non-limiting embodiments or aspects, historical test data may include recorded test failures and recorded test executions for each test 204. Historical test data including recorded test failures may include the total number of test failures of test 204 captured over time during the lifecycle of test 204. Historical test data including recorded test executions may include the total number of test executions of test 204 captured over time during the lifecycle of test 204. Historical test data may be stored in database 216 or a storage component, making the test selection systems 100 and 200 accessible to the historical test data.

[0082] Continue to refer to Figure 3As shown at step 306, process 300 may include receiving a component criticality score and defect definition corresponding to source file 206. For example, test selection systems 100, 200 may receive corresponding component criticality scores based on source file 206 or multiple source files 206 corresponding to detected code changes. The component criticality score may be a score assigned to the component (e.g., software component, software module, software package, etc.) to which source file 206 belongs. Additionally or alternatively, the component criticality score may be a score assigned to source file 206 or multiple source files 206 identified in the corresponding detected code change. In some non-limiting embodiments or aspects, the component criticality score may be received from a user of test selection systems 100, 200 via input through an input component. Additionally or alternatively, the component criticality score may be received from a database 216 storing component criticality scores or other storage components.

[0083] Component criticality scoring can be based on a scoring algorithm. Component criticality scores can be based on the importance of source file 206 or the component, taking into account any relevant factors such as business use cases, system availability, component development costs, component requirements, etc. For example, a component or source file 206 might be assigned a component criticality score of 5 by the user (where 5 can indicate high importance of the component), where the component or source file is part of the application's user login component. In contrast, a component or source file might be assigned a component criticality score of 1 by the user (where 1 can indicate low importance of the component), where the component or source file is part of a settings component used to change the colors of the application's user interface. It should be understood that the scoring algorithms provided herein are exemplary, non-limiting embodiments provided for illustrative purposes, and the scoring algorithm can be any feasible scoring algorithm used to determine the component criticality score of a component. The scoring algorithm can include any such scale, calculation, formula, or representation of component criticality.

[0084] In some non-limiting embodiments or aspects, the defect definition corresponding to source file 206 associated with the detected code change may be based on a predefined score, rating, or definition of a problem found in the source file. Problems may include vulnerabilities, test failures, customer complaints, bug reports, etc. Problems may be stored in a problem tracking database (e.g., In DevOps Server, etc. Additionally, issues with source file 206 can be collected and stored over time. In some non-limiting embodiments or aspects, test selection systems 100, 200 can receive defect definitions by communicating with an issue tracking database. In some non-limiting embodiments or aspects, the issue may be related to the component to which source file 206 belongs. For example, when a code change is detected in source file 206 belonging to the user login component of an application, test selection systems 100, 200 can receive a defect definition from an issue tracking database related to the user login component, but the issue may not be directly related to the corresponding source file 206. In other non-limiting embodiments or aspects, the issue may be directly related to the corresponding source file 206 associated with the detected code change.

[0085] In some non-limiting embodiments or aspects, a defect definition may include a defect rating defined by a specific problem database being used. In some non-limiting embodiments or aspects, a defect definition may be defined by a scoring algorithm. For example, if in When tracking issues on other similar issue tracking platforms such as software repositories, defect definitions can include ratings such as P0 (Break), P1 (Critical), P2 (Default), P3 (Required), or P4 (e.g., a spelling error in a document). It should be understood that the scoring algorithms and defect definitions described herein can be applied to source document 206 or components of a software system. The following examples will refer to components. For example, if a component includes a P0 or P1 defect, it can be assigned a defect definition flag of 1 or "True". If a component does not include a P0 or P1 defect, it can be assigned a defect definition flag of 0 or "False". It should be understood that the scoring algorithms provided herein are exemplary, non-limiting embodiments provided for illustrative purposes, and the scoring algorithm can be any feasible scoring algorithm used to determine the defect definition of a component. The scoring algorithm can include any such scale, calculation, formula, or representation that measures the level of defect in a component. For example, a component with a P0 defect can be assigned a defect definition of 5 (where 5 can indicate a critical defect). In contrast, a component with a P4 defect can be assigned a defect definition of 1 (where 1 can indicate a minor defect). In other issue tracking systems, defects can be assigned severity ratings, risk ratings, priority ratings, etc. These ratings can be used to define defect definitions for components. In some non-limiting embodiments, defect definitions for components can be captured within a defined time period. For example, defect definitions can be assigned to components for issues that occurred within the past year or several months. When test selection systems 100 and 200 receive a defect definition, the issue can be defined as resolved or unresolved in the issue tracking database. Additionally or alternatively, the issue for which the defect definition is defined can be related to or unrelated to code changes detected by test selection systems 100 and 200.

[0086] Continue to refer to Figure 3 As shown at step 308, process 300 may include generating a key value for each entry in test map 212 based on the defect score of test 204, the component criticality score of component or source file 206 corresponding to the detected code change, and the defect definition corresponding to the detected code change. For example, test selection systems 100, 200 may generate a key value for an entry in test map 212 corresponding to a detected code change by summing the defect score, component criticality score, and defect definition. In this case, a higher key value may indicate that the component or source file 206 corresponding to the detected code change is of lower quality but may be of higher importance. Therefore, the detected code change may require more tests 204 to be performed. In contrast, a lower key value may indicate that the component or source file 206 corresponding to the detected code change is of higher quality but may be of lower importance. Therefore, the detected code change may require fewer tests 204 to be performed. It should be understood that the scoring algorithm provided herein is an exemplary, non-limiting embodiment provided for illustrative purposes, and the scoring algorithm may be any feasible scoring algorithm used to determine the key value of a component. The scoring algorithm may include any such scale, calculation, formula, or representation, etc., that measures the key value of the entries in test map 212. For example, the key value may be determined by averaging the defect score of the detected code change, the component criticality score, and the defect definition. In some non-limiting embodiments or aspects, a higher key value may indicate high quality and low importance, while a lower key value may indicate low quality and high importance.

[0087] In some non-limiting embodiments or aspects, a machine learning model 214 may be used to generate key values, the machine learning model receiving and processing defect scores, component criticality scores, and defect definitions corresponding to test-to-source file entries in test map 212 as input. In some non-limiting embodiments or aspects, the machine learning model 214 may use stored historical test data from previously detected code changes (e.g., previously generated test map 212) as input to enhance its learning and influence key value output. Alternatively, the machine learning model 214 may use historical test data (e.g., test results 220 from previously executed test 204) as input to enhance its learning capabilities and influence key value output.

[0088] Continue to refer to Figure 3As shown in step 310, process 300 may include determining a subset of tests 204 based on key values ​​corresponding to entries in test map 212. In some non-limiting embodiments or aspects, machine learning model 214 may be used to determine a subset of tests 204 based on key values ​​corresponding to entries in test map 212. Machine learning model 214 may generate a recommended subset of tests 204 based on key values ​​and predefined test categories. For example, tests 204 may be categorized into categories S1, S2, and S3. For test map 212 corresponding to detected code changes, multiple key value scores may be generated. For example, an entry in test map 212 containing a test-to-source file mapping may be assigned a key value. Test-to-source file mappings may have low key value scores, which may indicate higher quality or lower importance. Recommended tests 204 may then be selected from a single category S1. In contrast, if the key value scores indicate low quality or higher importance, recommended tests 204 may be selected from categories S1, S2, and S3 to maximize code coverage. Recommended test 204 can be recorded in mapping file 218 via machine learning model 214, along with the corresponding test category and / or source file 206. The number of tests, the identity of test 204, or the category of test included in mapping file 218 should not be construed as being limited by the test 204 included in test mapping 212. In other words, machine learning model 214 can recommend more, fewer, or the same test 204 recorded in test mapping 212 based on the model performance of machine learning model 214.

[0089] Continue to refer to Figure 3 As shown in step 312, process 300 may include executing a subset of tests along with the executable software project. In some non-limiting embodiments or aspects, software repository 202 may be compiled and built into the executable software project. A subset of tests 204 may be formatted and provided to a computing device in an executable format to be executed along with the executable software project. The executable software project and the executable subset of tests 204 may be executed simultaneously to test detected code changes. In some non-limiting embodiments or aspects, once the subset of tests 204 has been executed along with the executable software project, a test report including test results 220 may be generated based on the execution of the subset of tests 204.

[0090] Now for reference Figure 4 , Figure 4This is a diagram illustrating exemplary components of device 400. Device 400 may correspond to one or more devices of test selection system 102, one or more devices of machine learning model 104, and / or one or more devices of software repository 106. In some non-limiting embodiments or aspects, test selection system 102, machine learning models 104, 214, and / or software repositories 106, 202 may include at least one device 400 and / or at least one component of device 400. Figure 4 As shown, the device 400 may include a bus 402, a processor 404, a memory 406, a storage unit 408, an input unit 410, an output unit 412, and a communication interface 414.

[0091] Bus 402 may include components that enable communication between parts of device 400. In some non-limiting embodiments or aspects, processor 404 may be implemented in hardware, software, firmware, and / or any combination thereof. For example, processor 404 may include a processor (e.g., a central processing unit (CPU), graphics processing unit (GPU), accelerated processing unit (APU), etc.), a microprocessor, a digital signal processor (DSP), and / or any processing component that can be programmed to perform a function (e.g., a field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), etc.). Memory 406 may include random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, optical memory, etc.) that stores information and / or instructions for use by processor 404.

[0092] Storage component 408 may store information and / or software related to the operation and use of device 400. For example, storage component 408 may include hard disk (e.g., magnetic disk, optical disk, magneto-optical disk, solid-state disk, etc.), compressed optical disk (CD), digital versatile optical disk (DVD), floppy disk, cassette tape, magnetic tape and / or another type of computer-readable medium, and corresponding drives.

[0093] Input component 410 may include a component that allows device 400 to receive information, such as via user input (e.g., a touchscreen display, keyboard, keypad, mouse, button, switch, microphone, camera, etc.). Alternatively, input component 410 may include sensors for sensing information (e.g., a Global Positioning System (GPS) component, accelerometer, gyroscope, actuator, etc.). Output component 412 may include a component that provides output information from device 400 (e.g., a display, speaker, one or more light-emitting diodes (LEDs), etc.).

[0094] Communication interface 414 may include transceiver components (e.g., transceiver, separate receiver and transmitter, etc.) that enable device 400 to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communication interface 414 may allow device 400 to receive information from another device and / or provide information to another device. For example, communication interface 414 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, etc. interface, interface, Interfaces, cellular network interfaces, etc.

[0095] Apparatus 400 can perform one or more processes described herein. Apparatus 400 can perform these processes based on software instructions stored in a computer-readable medium such as memory 406 and / or storage unit 408, executed by processor 404. Computer-readable medium (e.g., non-transient computer-readable medium) is defined herein as a non-transient memory device. A non-transient memory device includes memory space located within a single physical memory device or memory space distributed across multiple physical memory devices.

[0096] Software instructions can be read from another computer-readable medium or from another device into memory 406 and / or storage unit 408 via communication interface 414. When executed, the software instructions stored in memory 406 and / or storage unit 408 cause processor 404 to perform one or more processes described herein. Alternatively or additionally, hard-wired circuitry may be used in place of or in combination with the software instructions to perform one or more processes described herein. Therefore, the embodiments or aspects described herein are not limited to any particular combination of hardware circuitry and software.

[0097] supply Figure 4 The number and arrangement of components shown are for illustrative purposes only. In some non-limiting embodiments or aspects, with Figure 4 Compared to the components shown, device 400 may include additional components, fewer components, different components, or components arranged in a different manner. Alternatively, a set of components of device 400 (e.g., one or more components) may perform one or more functions described as being performed by another set of components of device 400.

[0098] Now for reference Figure 5 , Figure 5 This is a schematic diagram of components of a test selection system 500 for automatically selecting tests for regression testing of a software system using machine learning, according to some non-limiting embodiments or aspects. For example... Figure 5As shown, the test selection system 500 may include a test file 504 and a source file 506. The test file 504 may be the same as or similar to test 204, and the source file 506 may be the same as or similar to source file 206. In some non-limiting embodiments or aspects, the test selection system 500 may include a controller 508. The controller 508 may be the same as or similar to the test selection system 102. The controller 508 may be able to control the execution flow and data sharing between the test file 504, the source file 506, the intelligent engine 514, and the software repository 502.

[0099] In some non-limiting embodiments or aspects, the test selection system 500 may include an intelligent engine 514. The intelligent engine 514 may be the same as or similar to machine learning models 104, 214. The intelligent engine 514 may include one or more components, such as a decision module, a mapping module, and a listener, among others, of the machine learning models 104, 214. In some non-limiting embodiments or aspects, the intelligent engine 514 may generate a data file that may be the same as or similar to the test mapping 212 from a list of source files 506 and a list of test files 504. The decision module may be the same as or similar to machine learning model 214. The decision module may receive the data file as input and may generate a key value based on a defect score, component criticality score, and defect definition corresponding to a specific test 504. In some non-limiting embodiments or aspects, the decision module may generate a recommended subset of tests based on the key value corresponding to the test 504 and predefined test categories. The decision module may send the recommended subset of tests to the mapping module to create a mapping file.

[0100] In some non-limiting embodiments or aspects, the intelligent engine 514 may include a mapping module. The mapping module may receive a recommended test subset from the decision module. In some non-limiting embodiments or aspects, the mapping module may generate a mapping file based on the recommended test subset and source file 506. The mapping file may be the same as or similar to mapping file 218. In some non-limiting embodiments or aspects, the mapping file may be sent to controller 508 to obtain tests 504 of the test subset included in the mapping file.

[0101] In some non-limiting embodiments or aspects, the test selection system 500 may include a software repository 502. The software repository 502 may include a version control system or operate in conjunction with such a version control system. For example, as Figure 5 As shown, software repository 502 can be combined with Operation. In some non-limiting embodiments or aspects, software repository 502 may include continuous integration management tools (automation server, continuous integration server, etc.). Software repository 502 may communicate with controller 508. For example, software repository 502 may transmit a list of modified source files 506 to controller 508. The list of modified source files 506 may be the same as or similar to the list of source files 210.

[0102] In some non-limiting embodiments or aspects, the test selection system 500 can execute a subset of tests included in a mapping file together with the executable software project of the software repository 502. In some non-limiting embodiments or aspects, the software repository 502 can be compiled and built into the executable software project. A subset of tests 504 can be formatted by the mapping module and provided to the controller 508 in an executable format to be executed together with the executable software project. The executable software project and the executable subset of tests 504 can be executed simultaneously to test detected code changes. In some non-limiting embodiments or aspects, once the subset of tests 504 has completed execution together with the executable software project, a test report can be generated based on the execution of the subset of tests 504.

[0103] In some non-limiting embodiments or aspects, the test report may be transmitted to a listener of the intelligent engine 514 as input to the decision module for further processing. In some non-limiting embodiments or aspects, the test report may be stored for future use by the decision module or other machine learning models 214.

[0104] Although the disclosed subject matter has been described in detail for illustrative purposes based on embodiments or aspects currently considered most practical and preferred, it should be understood that such details are for the purposes described only, and the disclosed subject matter is not limited to the disclosed embodiments or aspects, but rather is intended to cover modifications and equivalent arrangements within the spirit and scope of the appended claims. For example, it should be understood that the currently disclosed subject matter is intended to be combined, as far as possible, with one or more features of any embodiment or aspect in combination with one or more features of any other embodiment or aspect.

Claims

1. A computer-implemented test selection method, comprising: generating, with at least one processor and based on a test list and a source file list, a test mapping comprising a mapping between a plurality of tests and a plurality of source files, wherein the plurality of tests and the plurality of source files are associated with a software repository; each source file in the test mapping is associated with a code change, wherein at least one first source file in the test mapping is associated with a code change of the at least one first source file, and wherein at least one second source file in the test mapping is associated with a code change of a source file having at least one common component with the at least one first source file; determining, with the at least one processor and at least one machine learning model, a defect score for each of the plurality of tests based on historical test data of the tests, wherein the defect score is based on a measure of defects of a test of the plurality of tests; receiving, with the at least one processor, a component criticality score and a defect definition for each of the plurality of source files, the defect definition comprising at least one of a pre-defined score or rating based on one or more issues in the source file, the one or more issues stored in an issue tracking database, wherein the component criticality score is based on a measure of component criticality of a source file of the plurality of source files; generating, with the at least one processor, a key value for each of the plurality of tests, the key value based on values of the defect score, the component criticality score, and the defect definition; determining, with the at least one processor and the at least one machine learning model, a subset of tests of the plurality of tests based on key values corresponding to the plurality of tests; and executing, with the at least one processor, the subset of tests with the software repository.

2. The computer-implemented test selection method of claim 1, further comprising: generating, with the at least one processor, test results based on execution of the subset of tests; and inputting the test results to the at least one machine learning model to further train the at least one machine learning model. generating the test mapping comprises:

3. The computer-implemented test selection method of claim 1, wherein, detecting at least one first source file based on a change to the at least one first source file, wherein the change comprises a difference between a first version of the at least one first source file and a second version of the at least one first source file; selecting at least one test corresponding to the at least one first source file based on a pre-defined mapping of source files to tests; compiling the source file list and the test list, wherein the source file list comprises a source file name of the at least one first source file and the test list comprises a test file name of the at least one test; and generating the test mapping from the source file list and the test list by mapping the source file name to its corresponding test file name based on the pre-defined mapping of source files to tests. ​ 4. The computer- implemented test selection method of claim 3, wherein, The at least one first source file is a new source file and the change includes the new source file added to the software repository.

5. The computer- implemented test selection method of claim 1, wherein, determining the test subset includes: generating a recommended test subset based on the key values and predefined test categories; generating a mapping file based on the recommended test subset and the source files; storing the mapping file in a database; and formatting the recommended test subset into an executable format.

6. The computer- implemented test selection method of claim 1, wherein, The at least one machine learning model uses supervised learning and predictive determination of the test subset.

7. The computer- implemented test selection method of claim 5, wherein, The mapping file stored in the database is used as an input to the at least one machine learning model for further training.

8. A test selection system comprising: at least one machine learning model; a software repository; and at least one processor, wherein the at least one processor is programmed or configured to: generate a test mapping based on a test list and a source file list, the test mapping including a mapping between a plurality of tests and a plurality of source files, wherein the plurality of tests and the plurality of source files are associated with the software repository; each source file in the test mapping is associated with a code change, wherein at least one first source file in the test mapping is associated with a code change of the at least one first source file, and wherein at least one second source file in the test mapping is associated with a code change of a source file having at least one common component with the at least one first source file; determine, using the at least one machine learning model, a defect score for each of the plurality of tests based on historical test data for the tests, wherein the defect score is based on a measure of defects for a test of the plurality of tests; receive a component criticality score and a defect definition for each of the plurality of source files, the defect definition including at least one score based on data associated with the source file in an issue tracking database, wherein the component criticality score is based on a measure of component criticality for a source file of the plurality of source files; generate a key value for each of the plurality of tests, the key value based on values of the defect score, the component criticality score, and the defect definition; determine, using at least one machine learning model, a test subset of the plurality of tests based on the key values corresponding to the plurality of tests; and execute the test subset with the software repository.

9. The system of claim 8, wherein the at least one processor is further programmed or configured to: generate test results based on execution of the test subset; and input the test results to the at least one machine learning model to further train the at least one machine learning model.

10. The system of claim 8, wherein, when generating the test mapping, the at least one processor is programmed or configured to: detect at least one first source file of the plurality of source files based on a change to the at least one first source file, wherein the change includes a difference between a first version of the at least one first source file and a second version of the at least one first source file; selecting, from the plurality of tests, at least one test corresponding to the at least one first source file based on a predefined mapping of source files to tests; compiling the list of source files and the list of tests, wherein the list of source files includes a source file name of the at least one first source file and the list of tests includes a test file name of the at least one test; and generating the test mapping from the list of source files and the list of tests based on the predefined mapping of source files to tests by mapping the source file name to its corresponding test file name.

11. The system of claim 10, wherein, the at least one first source file is a new source file and the change includes the new source file added to the software repository.

12. The system of claim 8, wherein, when determining the subset of tests, the at least one processor is programmed or configured to: generate a recommended subset of tests based on the key value and a predefined test category; generate a mapping file based on the recommended subset of tests and corresponding source files; store the mapping file in a database; and format the recommended subset of tests into an executable format.

13. The system of claim 8, wherein, the at least one machine learning model uses supervised learning and predictive determination of the subset of tests.

14. The system of claim 12, wherein, the mapping file stored in the database is used as an input to the at least one machine learning model for further training.

15. A computer program product, the computer program product comprising at least one non-transitory computer-readable medium including one or more instructions that, when executed by at least one processor, cause the at least one processor to: generating a test map based on the test list and the source file list, the test map including a mapping between the plurality of tests and the plurality of source files, wherein, the plurality of tests and the plurality of source files are associated with a software repository; each source file in the test mapping is associated with a code change, wherein at least one first source file in the test mapping is associated with a code change of the at least one first source file, and wherein at least one second source file in the test mapping is associated with a code change of a source file having at least one common component with the at least one first source file; determine a defect score for each of the plurality of tests based on historical test data of the test, wherein the defect score is based on a measure of defects of a test of the plurality of tests; receive a component criticality score and a defect definition for each of the plurality of source files; the defect definition includes at least one score, wherein the component criticality score is based on a measure of component criticality of a source file of the plurality of source files; generate a key value for each of the plurality of tests, the key value based on values of the defect score, the component criticality score, and the defect definition; determine a subset of tests of the plurality of tests based on the key values corresponding to the plurality of tests; and execute the subset of tests with the software repository.

16. The computer program product of claim 15, wherein the one or more instructions further cause the at least one processor to: generate test results based on execution of the subset of tests; and inputting the test results to at least one machine learning model to further train the at least one machine learning model.

17. The computer program product of claim 15, wherein, The one or more instructions, when causing the generation of the test mapping, further cause the at least one processor to: detect the at least one first source file based on a change to the at least one first source file, wherein the change comprises a difference between a first version of the at least one first source file and a second version of the at least one first source file; select at least one test corresponding to the at least one first source file based on a predefined mapping of source files to tests; compile a source file list and a test list, wherein the source file list comprises a source file name of the at least one first source file and the test list comprises a test file name of the at least one test; and generate the test mapping from the source file list and the test list by mapping the source file name to its corresponding test file name based on the predefined mapping of source files to tests.

18. The computer program product of claim 17, wherein, The at least one first source file is a new source file and the change comprises the new source file added to the software repository.

19. The computer program product of claim 15, wherein, The one or more instructions, when causing the determination of the subset of tests, further cause the at least one processor to: generate a recommended subset of tests based on the key value and a predefined test category; generate a mapping file based on the recommended subset of tests and corresponding source files; store the mapping file in a database; and format the recommended subset of tests into an executable format.

20. The computer program product of claim 19, wherein, The mapping file stored in the database is used as input to at least one machine learning model for further training.

Images