Generate test models from BDD scenarios based on BDD step definitions and similarity analysis using neuro-linguistic programming and machine learning mechanisms

Abstract

A test model is generated from a behavior-driven development scenario based on behavior-driven development step definitions and similarity analysis using neuro-linguistic programming and machine learning mechanisms. The present invention relates to a method for automatically verifying a software program in a behavior-driven development (BDD) environment and a data processing system configured to perform such a method. Individual test steps of a BDD test scenario are first matched and then assigned to an existing test step definition from a BDD framework. If a one-to-one match is not possible, natural language processing (NLP) is used to decide whether the assignment is possible with a certain probability of matching. The assigned test step definition is used to generate a graphical test model, such as a UML diagram, for the test scenario. Finally, an executable test script is generated to test the software program. The present invention particularly relates to behavior-driven development (BDD) and combines traditional BDD advantages with model-based testing (MBT) for improved convenience and automation in the case of complex software packages. Automated step matching allows efficient mapping of BDD step phrases to a test automation framework and supports structured development of necessary framework code. The graphical test model adds an additional layer of abstraction and provides an opportunity to check the consistency and completeness of the BDD scenario.

Description

Technical Field

[0001] The present invention relates to a computer-implemented method for automatically verifying a software program in a behavior driven development environment. The invention further relates to a device comprising a processor configured to perform such a method. The invention particularly relates to behavior driven development (BDD). Background Art

[0002] In recent years, BDD has emerged as an agile software development methodology for specifying and executing automated acceptance tests of software programs. BDD was introduced by Dan North in 2006 to simplify test-driven development (TDD), see for example Brandel et al., "Drei Methoden, ein Ziel: Testautomatisierung mit BDD, MBT und KDT imVergleich", Softwaretechnik-Trends, 35(3), 2015. TDD is a software development methodology that essentially stipulates that for each unit of software, the software developer must first define a specific set of tests for that unit, then implement the unit, and finally verify that the implementation of the unit succeeds in testing. BDD combines test-driven development (TDD), object-oriented analysis (OOA), object-oriented design (OOD), and domain-driven design (DDD) to provide a unified language and method for handling such a software development process from requirements analysis to implementation.

[0003] BDD is largely facilitated by using a simple domain-specific language (DSL) that uses natural language constructs (e.g., English-like sentences) that can express the behavior and expected results of the software. This "universal language" can be understood and used by quality managers, domain experts, software developers, and customers. BDD uses a semi-formal format for the behavioral specification of software, which is borrowed from the specification of user stories in the field of object-oriented analysis and design.

[0004] To this end, each software unit is decomposed into so-called scenarios, each of which tests a separate aspect of the software. Each scenario is in turn divided into test steps that describe the expected results of the corresponding aspect of the software starting from a given initial condition and running through predefined events. Each scenario and its test steps are formulated as a natural language script, which can subsequently be converted into an executable test script in an automated manner. The executable test script can then be executed as an automated test for testing the correct implementation of the software. The software requirements within the test script are typically written in "given-when-then" statements based on a common language for domain-driven design. This is intended to facilitate the conversion between the language used to define domain-driven requirements and the programming language used to implement them.

[0005] A widely used test automation framework for automated acceptance tests written in a BDD style is called Cucumber, which includes a parser for a concise language called Gherkin. The desired behavior of the software is declaratively specified in Gherkin:

[0006] -GIVEN (precondition / initial condition) ...

[0007] -WHEN (event / action / trigger) …

[0008] -THEN (effect / system response to be observed)….

[0009] Such a descriptive language is semi-formal, where capitalized words (GIVEN, WHEN, THEN) act as pre-specified keywords. Due to the simple syntax and natural language keywords, technical testers can understand and manually execute BDD requirements. Cucumber runs through these keywords and processes them step by step, mapping each non-capitalized phrase following these keywords to a parameterized function call. Traditionally, Ruby scripts are used for this purpose in Cucumber, which replaces test steps by automating program calls and thereby making BDD descriptions automatically executable. However, Cucumber now supports a variety of different programming languages ​​through various implementations, including Java and C#.

[0010] BDD is well understood and easy to implement. However, in large and complex use cases, this approach and its textual and manually created scenarios may lack manageability to handle a large number of scenarios and complex test sets while ensuring completeness and consistency. For the development of complex systems, approaches like model-based testing (MBT) or keyword-based testing (KBT) are often considered more appropriate. In particular, the MBT approach allows the use of visual representations of scenarios, such as diagrams in the Unified Modeling Language (UML), to review and verify the completeness and consistency of even complex test scenarios. However, MBT must be embedded into the existing development and testing process of each software component individually. Summary of the invention

[0011] In this context, it is an object of the present invention to find a solution with improved convenience and automation for the validation of complex software packages.

[0012] According to one aspect of the present invention, a computer-implemented method for automatically verifying a software program in a behavior-driven development environment includes: receiving test scenarios using a data processing system, each test scenario defining the expected behavior of a software program in consecutive test steps, the test steps being formulated in a domain-specific language using natural language phrases and describing the expected results of the software program for predefined events based on given initial conditions; importing test step definitions from a behavior-driven development environment; determining, for each test step of the test scenario, whether the test step matches one of the test step definitions based on the natural language phrases of the test step; assigning all matching test steps to corresponding test step definitions; applying natural language processing (NLP) to the natural language phrases of any test steps that remain unmatched, wherein NLP provides a confidence level for each unmatched test step to correspond to one of the test step definitions; assigning any unmatched test step to a corresponding test step definition when the confidence level exceeds a first predefined matching probability; and at least one of the following: generating a graphical test model for the test scenario based on the assigned test step definition; and generating an executable test script for the test scenario based on the assigned test step definition.

[0013] According to another aspect of the invention, a data processing system comprises a processor configured to execute the method according to the invention.

[0014] According to a further aspect of the invention, a computer program comprises executable program instructions which, when executed, are configured to carry out the method according to the invention.

[0015] According to yet another aspect of the present invention, a non-transitory computer-readable data storage medium includes executable program instructions configured to carry out a method according to the present invention when executed.

[0016] The non-transitory computer-readable data storage medium may include or consist of any type of computer memory, in particular semiconductor memory such as solid-state memory. The data storage medium may also include or consist of a CD, a DVD, a Blu-ray disc, a USB memory stick, a memory card (e.g., an SD card), etc.

[0017] According to a further aspect, the invention provides a data stream representing or configured to generate executable program instructions configured to carry out a method according to the invention when executed.

[0018] One idea of ​​the present invention is to provide a means of leveraging the benefits of BDD with easy-to-use and natural language-based scenarios while maintaining the manageability required for large and complex development projects. To this end, the proposed solution automatically assigns an existing test step definition of a test automation framework (from an integrated BDD development environment) to the test step of each scenario. If a literal and / or unambiguous match is not possible - for example because the corresponding scenarios are written in a different style and / or use different wording - an NLP algorithm is used to find the best match for the corresponding test step among the existing test step definitions. If the probability of this best match is high enough to provide a correct / likely fit between the test step and the test step definition, for example, if it has a matching probability of at least 80% or 90% or more, the test step is assigned to the corresponding test step definition.

[0019] This method allows BDD step phrases to be effectively mapped to a test automation framework that integrates a BDD development environment, and supports the structured development of the necessary framework code. Further, it facilitates the automatic generation and synchronization of graphical test models from BDD scenarios, so that even for large and complex development projects, the advantages of both BDD and MBT methods can be used. Graphic test models can be used to visualize, review, and modify test cases, thereby ensuring the consistency and completeness of BDD scenarios. For example, missing scenarios can be identified based on test model reviews. As another example, similar scenarios can be combined into a single scenario. Further, the ability to use MBT technology adds additional levels of abstraction and supports change management. Executable test scripts can be automatically generated directly based on assigned test step definitions and / or after validating (one or more) scenarios based on the generated test model.

[0020] Advantageous embodiments and improvements of the invention are found in the dependent claims. According to one embodiment, the method may further comprise updating the respective test step definition based on the natural word phrase of the respective test step when the confidence level is above a first predefined match probability. Thus, an existing phrase pattern definition from a BDD test automation framework may be adapted to include an alternative and / or modified test step definition corresponding to the matched test step. The first predefined match probability may be set to a high confidence value of 80% or higher, such that there is a high match probability between the test step and the test step definition.

[0021] According to one embodiment, the method may further include adding a test step definition to the behavior driven development environment corresponding to the corresponding test step when the confidence level is lower than the second predefined matching probability. Therefore, in the case where the confidence level is lower than the reference probability (the reference probability may be, for example, 50% or a similar probability), it is determined that the test step does not match any existing test step definition. Alternatively, the test step is used to define a new test step definition, which is then added to the BDD test automation framework and can be further used.

[0022] According to one embodiment, if the confidence level is lower than the first predefined match probability but higher than the second predefined match probability, user verification may be requested. For example, the first predefined match probability may be set to 80% or 90%, and the second predefined match probability may be set to 50%. If the confidence level is higher than the first predetermined match probability, the test step is considered to match the corresponding test step definition, which may then be updated based on the formula of the test step. If the confidence level is lower than the second predefined match probability, the test step does not match any existing definition and may therefore be used to define a new definition. However, in the intermediate range between 50% and 80% (or 90%), the situation may not be clear, i.e., the test step may or may not match one of the existing definitions. In this case, user input may be required to resolve the further process, i.e., whether to introduce a new definition, whether to update an existing definition, or whether to discard the scenario, etc.

[0023] According to one embodiment, the method may further include feeding the user verification to the machine learning algorithm of the NLP. For example, the commonality criteria of the NLP can be adjusted based on the verification results, such as reducing the relevance of certain phrases, identifying invalid commonalities and / or commonalities that have not yet been identified. The commonality detection accuracy can then be compared in future executions to check whether the optimized criteria improves the accuracy of the NLP. In the long run, this can reduce the workload of manual verification and improve the accuracy of commonality detection over time. After the required training and optimization of the NLP engine, the algorithm of the present invention can detect and match phrases in a completely unattended and automated manner.

[0024] According to one embodiment, generating the graphical test model may include combining similar test scenarios based on test steps assigned to the same test step definition.

[0025] According to one embodiment, generating the graphical test model may include identifying test data within the test scenario based on natural language phrases.

[0026] According to one embodiment, the graphical test model may include a Unified Modeling Language diagram, ie, a UML diagram.

[0027] According to one embodiment, the method may further include comparing the graphical test model with the test scenarios to determine whether the graphical test model conforms to the expected behavior of the software program. Thus, based on the generated graphical test model, missing and / or incorrect scenarios may be identified.

[0028] The invention will be explained in more detail with reference to exemplary embodiments depicted in the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are included to provide a further understanding of the present invention and are incorporated into and constitute a part of this specification. The drawings illustrate embodiments of the present invention and together with the description are used to explain the principles of the present invention. Other embodiments of the present invention and many of the expected advantages of the present invention will be readily appreciated as they become better understood by reference to the detailed description below. The elements of the drawings are not necessarily to scale relative to each other. In the drawings, unless otherwise indicated, the same reference numerals designate the same or functionally identical components:

[0030] Figure 1 A device having a processor that performs a method according to an embodiment of the present invention is shown;

[0031] Figure 2 Shows a demo Figure 1 Schematic flow charts of various aspects of the method;

[0032] Figure 3 Shows a demo Figure 1 Schematic flow charts of various aspects of the method;

[0033] Figure 4 Shows the use of Figure 1 Example of a graphical test model derived using the method. DETAILED DESCRIPTION

[0034] Although specific embodiments have been illustrated and described herein, those of ordinary skill in the art will appreciate that various alternative and / or equivalent implementations may be substituted for the specific embodiments shown and described without departing from the scope of the present invention. In general, this application is intended to cover any adaptation or variation of the specific embodiments discussed herein.

[0035] Figure 1 A data processing system 10 is shown having a processor 11 that executes a method M according to an embodiment of the present invention. Certain aspects of method M are illustrated in Figure 2 and 3 by way of example.

[0036] In addition to the processor 11, the data processing system 10 may also include conventional components such as accessible memory, storage units, input units, output units, etc. (not shown). The processing unit 10 used herein refers to any type of computer or computing circuit, such as but not limited to a microprocessor unit, a microcontroller, a graphics processing unit, a digital signal processing unit, or any other type of processing circuit.

[0037] Method M provides automatic verification of software programs in a behavior-driven development environment, such as an integrated development environment, which may include a BDD test automation framework like Cucumber, SpecFlow, Behave, or similar with libraries having BDD scripts and phrase patterns / test step definitions.

[0038] Method M includes receiving test scenarios 1 under M0 using the data processing system 10, for example, by importing them from a BDD development environment. Each test scenario 1 defines the expected behavior of the software program in successive test steps 2. The test scenarios 1 and thus the test steps 2 are formulated in a domain-specific language using natural language phrases and describe the expected results ("THEN") of the software program for predefined events ("WHEN") based on given initial conditions ("GIVEN"). The test scenarios 1 thus represent the specification and / or requirements of the software program in chronological order.

[0039] As a simple example, a registration / login software may include the following (schematic) scenarios, where the keywords GIVEN, WHEN, THEN each define a corresponding test step 2:

[0040] - Scenario 1: Login successful

[0041] GIVEN The user has entered valid credentials

[0042] WHEN Click login

[0043] THEN The start screen is shown

[0044] -Scenario 2: Wrong password

[0045] GIVEN The user has entered invalid credentials

[0046] WHEN Press Login

[0047] THEN an error message is shown

[0048] -Scenario 3: Unregistered User

[0049] GIVEN Unregistered user has entered some credentials

[0050] WHEN Press Login

[0051] THEN an error message is shown

[0052] -Scene 4: Registration

[0053] GIVENUnregistered User

[0054] WHEN Click to register

[0055] THEN the registration dialog is shown.

[0056] Method M further includes importing a test step definition 3 from a behavior driven development environment under M1. For the above example, such an existing test step definition 3 may be as follows (formulated in any programming language, such as Java or C#):

[0057] -@Given ("^User has entered [*credentials]$")

[0058] public void

[0059] enter_credentials (UCred argl)

[0060] -@When ("^Click the login button$")

[0061] @When ("Press Login$")

[0062] public void click_on_login ()

[0063] -@When ("^Click the registration button$")

[0064] public void

[0065] click_on_registration()

[0066] -@Then ("^Show start screen$")

[0067] public void

[0068] verify_start_screen_shown()

[0069] -@Then ("^show error message$")

[0070] public void

[0071] verify_error_msg_shown()

[0072] -@Then ("^Show the registration screen$")

[0073] public void

[0074] verify_registration_shown()

[0075] Next, method M includes determining, under M2, for each test step 2 of test scenario 1, whether test step 2 matches one of the natural language phrases based on test step 2 and test step definitions 3. Method M further includes assigning all matching test steps 2 to corresponding test step definitions 3 under M3.

[0076] In the example above, the matching steps may include:

[0077] - GIVEN User has entered valid credentials

[0078] GIVEN The user has entered invalid credentials

[0079] @Given("^User has entered [* credentials]$")

[0080] - WHEN Click to log in

[0081] WHEN Press Login

[0082] @When ("^Click the login button$")

[0083] @When ("^Press Login$")

[0084] However, a one-to-one match on the text may not be achieved for all test steps 2. For example, "WHEN is pressing login" is different from "WHEN press login" due to the different usage of the word "press". However, these two test steps 2 are similar, and therefore natural language processing (NLP) can be used to identify these similarities. Similarly, "GIVEN unregistered user has entered some credentials" is similar to:

[0085] @Given("^User has entered [*credentials]$")

[0086] And "WHEN click to register" is similar to:

[0087] @When ("^Click the register button$").

[0088] To identify these similarities, method M further includes applying NLP at M4 to the natural language phrases of any test step 2 that remain unmatched. The NLP provides a confidence level for each unmatched test step 2 to correspond to one of the test step definitions 3. Method M further includes assigning any unmatched test step 2 to a corresponding test step definition 3 when the confidence level exceeds a first predefined match probability at M5.

[0089] Figure 3 An example is shown, where once one of the test steps 2 cannot be matched under steps M2, M3, the NLP is called under M4. The NLP provides a confidence level, which is then compared to two predefined matching probabilities, a high probability of 90% and a low probability of 50%. If the confidence level is above 90%, the corresponding test step 2 is dispatched to the corresponding test step definition 3 under M5. The test step definition 3 can be updated by including the dispatched test step 2 in the repository of test step definitions 3 within the BDD framework (see Figure 2 In the case where the confidence level is lower than 50%, a new test step definition 3 is added to the BDD test automation framework corresponding to the test step 2 that does not yet exist (see Figure 2 Reference symbol T2 in ).

[0090] If the confidence level is between 50% and 90%, request user verification (see Figure 1 ), which is then used as input to the NLP's machine learning algorithm at T3 to improve the accuracy of the NLP in future runs. For example, the results of the assignments can be analyzed for unidentified commonalities, such as "select" should be equal to "click", or for invalid commonalities, such as "start screen" should be different from "registration screen". Further, the relevance of certain words or phrases may be reduced, such as in the case of irrelevant words like "some". By optimizing the NLP engine, manual intervention in the NLP application can be reduced and / or completely avoided in consecutive runs of method M.

[0091] In the example above, after running NLP, the dispatch for Test Step 2 and Test Step Definition 3 might look like this:

[0092] - GIVEN User has entered valid credentials

[0093] GIVEN The user has entered invalid credentials

[0094] GIVEN Unregistered user has entered some credentials

[0095] @Given("^User has entered [*credentials]$")

[0096] - WHEN Click to log in

[0097] WHEN Press Login

[0098] WHEN Pressing Login

[0099] @When ("^Click the login button$")

[0100] @When ("^Press Login$")

[0101] - WHEN Click to register

[0102] @When ("^Click the registration button$")

[0103] - THEN the start screen is shown

[0104] @Then ("^Show start screen$")

[0105] - THEN an error message is displayed

[0106] @Then ("^Show error message$")

[0107] - THEN the registration dialog is shown

[0108] @Then ("^Show registration dialog$")

[0109] Update test step definition 3 may include:

[0110] @When ("^Click the login button$")

[0111] @When ("^Press Login$")

[0112] @When ("Pressing login$")

[0113] public void click_on_login ()

[0114] Next, the method M includes generating a graphical test model 4 for the test scenario 1 based on the assigned test step definition 3 under M6. The graphical test model 4 can be represented by, for example, a unified modeling language diagram. Here, similar scenarios 1 can be combined based on the assignment of test steps 2 to test step definitions 3. Figure 4 An example is shown in , where two graphical test models 3 are generated from the above example, namely the case where the user has entered the credentials and the case where the user has not yet entered the credentials. The credentials may be invalid or valid, or the user may not have registered at all. These different test data (user credentials: invalid, unregistered, valid) can be identified by NLP based on natural language phrases and can be used to combine similar test scenarios 1, such as Figure 4 shown.

[0115] like Figure 2 As indicated in , method M may include, at T4, comparing the graphical test model 4 with the test scenario 1 to determine whether the graphical test model 4 conforms to the expected behavior of the software program. For example, based on these graphical test models 4, missing scenarios 1 may be identified, such as a registration attempt by a user with expired credentials or an already registered user. In addition, optimized test scenarios 1 may be generated, such as:

[0116] Scenario 1: Login successful

[0117] GIVEN User has entered [valid] credentials

[0118] WHEN Click the login button

[0119] THEN The start screen is shown

[0120] - Scenario 2: Wrong password

[0121] GIVEN User has entered [INVALID] credentials

[0122] WHEN Click the login button

[0123] THEN an error message is displayed

[0124] The method M further comprises generating, under M7, an executable test script 6 for the test scenario 1 based on the assigned test step definition 3. To this end, existing BDD tools and frameworks, such as Cucumber or similar tools and frameworks, may be utilized.

[0125] Thus, the approach M described above provides a means to leverage the benefits of BDD (with easy-to-use, natural language-based scenarios) while maintaining the manageability required for large, complex development projects. Generating and synchronizing test models from BDD scenarios allows leveraging the advantages from both BDD and MBT approaches, especially for large, complex development projects. Model-based review and generation of test cases ensures the consistency and completeness of BDD scenarios. The ability to add additional levels of abstraction using MBT techniques bridges the gap between RE-focused use of BDD and BDD-based test automation approaches. Automated step matching using machine learning allows efficient mapping of BDD step phrases to the test automation framework and supports structured development of the necessary framework code.

[0126] In the foregoing detailed description, various features are grouped in one or more examples or in examples for simplifying the present disclosure. It should be understood that the above description is intended to be illustrative, not restrictive. It is intended to cover all alternatives, modifications and equivalents. After reading the above description, many other examples should be apparent to those skilled in the art. The embodiments are selected and described in order to best explain the principles of the present invention and its practical application, so that other technical personnel in the art can best utilize the present invention and various embodiments with various modifications suitable for the intended specific use. After reading the above description, many other examples should be apparent to those skilled in the art.

Claims

1. A computer-implemented method (M) for automatically verifying a software program in a behavior-driven development environment, the method (M) include: receiving (M0) test scenarios (1) with a data processing system (10), each test scenario (1) defining an expected behavior of a software program in a series of test steps (2), the test steps being formulated in a domain specific language using natural language phrases and describing expected results of the software program for predefined events based on given initial conditions; Import (Ml) test step definitions from a behavior-driven development environment (3); determining (M2) for each test step (2) of the test scenario (1) whether the test step (2) matches one of the test step definitions (3) based on a natural language phrase of the test step (2); Assign (M3) all matching test steps (2) to the corresponding test step definitions (3); applying (M4) natural language processing (NLP) to the natural language phrases of any test steps (2) that remain unmatched, wherein the NLP provides a confidence level for each unmatched test step (2) to correspond to one of the test step definitions (3); When the confidence level exceeds a first predefined matching probability, assigning (M5) any unmatched test steps (2) to corresponding test step definitions (3); And at least one of the following: generating (M6) a graphical test model (4) for the test scenario (1) based on the assigned test step definition (3); and generating (M7) an executable test script (6) for the test scenario (1) based on the assigned test step definition (3); Further including: When the confidence level is higher than a first predefined matching probability, updating (T1) the corresponding test step definition (3) based on the natural word phrase of the corresponding test step (2); and When the confidence level is lower than a second predefined matching probability, adding (T2) a test step definition (3) to the behavior driven development environment corresponding to the corresponding test step (2).

2. The method (M) according to claim 1, in, If the confidence level is lower than the first predefined match probability but higher than the second predefined match probability, user authentication is requested.

3. The method (M) according to claim 2, further comprising: include: Feed the user verification (T3) to the NLP machine learning algorithm.

4. The method (M) according to any one of claims 1 to 3, in, Generating a graphical test model (4) includes combining similar test scenarios (1) based on test steps (2) assigned to the same test step definition (3).

5. The method (M) according to any one of claims 1 to 3, in, Generating a graphical test model (4) includes identifying test data (5) within the test scenario (1) based on natural language phrases.

6. The method (M) according to any one of claims 1 to 3, in, The graphical test model (4) includes a Unified Modeling Language diagram.

7. The method (M) according to any one of claims 1 to 3, further comprising: include: The graphical test model (4) is compared (T4) with the test scenario (1) to determine whether the graphical test model (4) conforms to the expected behavior of the software program.

8. A data processing system (10) comprising a processor (11) configured to perform the method (M) according to any one of claims 1 to 7.

9. A computer program product comprising executable program instructions configured to carry out the method (M) according to any one of claims 1 to 7 when being executed. 10 . A non-transitory computer-readable data storage medium comprising executable program instructions configured to, when executed, carry out the method (M) according to any one of claims 1 to 7 .

Images