Test reproduction guiding method based on test log analysis and man-machine interaction conversation

By combining computer vision and deep learning technology to establish a page jump model and using dialogue system to guide users to reproduce exception paths, the complexity of automated test logs is solved, the efficiency of tests and developers is improved, and the defect location and repair process is simplified. .

CN118916257APending Publication Date: 2024-11-08NANJING UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310511954.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-08
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The logs generated by automated testing tools are complex, making it difficult for testers to locate and analyze reproduction paths from them, making it difficult for developers to identify and fix exceptions.

Method used

Combining computer vision and deep learning technology, a page jump model is established, and users are guided to reproduce exception paths through dialogue systems, and a dialogue robot is used to provide guidance, simplifying log information analysis and path positioning.

Benefits of technology

Effectively parsing log information and automatically generating reproduction paths, improving the efficiency of testers and developers, and simplifying the defect location and repair process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916257A_ABST
    Figure CN118916257A_ABST
Patent Text Reader

Abstract

The invention discloses a test reproduction guiding method based on test log analysis and man-machine interaction conversation. A model conversion module, a monitoring module, a guiding module and a conversation robot module are included. And the model conversion module extracts page information from the GUI screenshot, merges pages according to the overall similarity of the structure and the content, and forms a page jump model according to a test step sequence. The monitoring module is used for monitoring reproduction operation of the user and putting the reproduction operation into the page model for comparison, and if deviation exists, the guiding module is called to correct deviation for the user according to a certain strategy. The dialogue robot module has basic interactive communication functions (such as reminding the user of deviating, inquiring whether the user needs help and inquiring whether reproduction succeeds) with the user, and can be matched with the guide module to remind the user until reproduction succeeds when the user seeks help.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of software testing, and in particular relates to test log analysis and test reproduction. After analyzing the provided test log and page screenshots, multiple strategies generate paths for reproducing defects and guide users to successfully reproduce through a dialogue system. Background Art

[0002] In recent years, the Internet has developed rapidly, and software has been related to all aspects of social life. Therefore, testing technology to ensure software quality has become extremely important. Traditional software testing technology can no longer meet the characteristics of diverse software versions and rapid iterations. Therefore, automated testing technology for software has become an important hot issue in academia and industry.

[0003] In order to cope with the diverse development of application functions, there are many available automated testing tools. After using these tools for automated testing, a large amount of information about the test content (such as test system, logs, etc.) will be obtained. The standard stream and error stream will record the steps and exception information that trigger the exception, which can effectively reduce the burden of manual testing. However, due to the complex and abstract information in the log, it is difficult for testers to directly locate the exception, analyze the recurrence path, and reproduce it with the naked eye, which makes it difficult for developers to identify the real defects. Take the Monkey framework as an example. After testing the application, it will output the standard stream and error stream logs in plain text. Because there are no page and component screenshots, testers must gradually understand the redundant operations and coordinates in the standard stream and parse the method body that triggers the exception information in the error stream, which consumes a lot of time and requires testers to understand the source code framework and development knowledge of the application.

[0004] Inspired by the above problems, the present invention deeply explores the rich and intuitive information contained in the interface images in the software, combines the existing automated testing tools with the graphical interface, and helps testers to parse log information and find the recurrence path. In addition, based on the dialogue system, the present invention provides anomaly recurrence guidance to help developers understand the causes of the triggering anomalies and successfully locate and repair them. Summary of the invention

[0005] The problem to be solved by the present invention is that the logs generated by automated testing tools are redundant and complex, and it is difficult for testers to locate the anomalies that need to be repaired and analyze the recurrence path. Our invention can understand the rich information in the test logs and automatically generate application jump relationships, obtain the recurrence path, and guide testers and developers to efficiently locate anomalies based on the dialogue system, which is convenient for future repairs.

[0006] The technical solution of the present invention is: a method for test reproduction guidance based on test log analysis and human-computer interactive dialogue, which is characterized by being able to convert the application jump model based on the log and other test information such as screenshots, and use the dialogue system to guide the user to reproduce after finding the shortest path. The method includes the following three modules:

[0007] 1) Model conversion module: The present invention defines the page state as a data structure composed of screenshots, controls, additional controls, pop-up windows and other information, which identifies a node in the jump graph. The page jump model uses computer vision and deep learning methods to establish and merge states. We first use computer vision methods to perform preliminary processing on the screenshots and classify similar pictures into one page. Then perform the following operations on all pages: first, use computer vision methods to capture pop-up windows and separate them into separate pages for processing; use computer vision methods to identify GUI structures, use deep learning methods to identify control content, calculate the overall similarity, and merge pages based on the similarity. Finally, connect the edges in the order of steps to form a page jump model.

[0008] 2) Monitoring module. The monitoring submodule will monitor and record the user's repeated operations and screenshots for a certain number of steps in real time, and compare them with the established page jump model. If the user performs an operation that is not in the existing record, the user can choose to ask the dialogue system for help. If a certain number of times is reached or the application is no longer in the recorded state, guidance will be provided proactively.

[0009] 3) Guidance module. Based on the page jump model, the guidance module will generate a path from the user's current page to the abnormal page according to a certain strategy, provide a guidance basis for the conversational robot, and assist the user in reproducing the abnormality. After the guidance is completed, the guidance module will ask the user whether to trigger the abnormality. If no selection is made, a new strategy will be selected for guidance. The path selection strategies are: the shortest path from the initial page to the abnormal page; the last 10 steps to the abnormality; take the last 10 steps to the abnormality as the original path, obtain all complete paths through depth-first search, and take the path with the longest common subsequence with the original path as the target path.

[0010] 4) Dialogue robot module: This module is the object that users interact directly with. In addition to basic interactive communication with users (such as reminding users of deviations, asking users if they need help, and asking whether the reproduction is successful), its core function is to receive the shortest path of the guidance module when the user seeks help, and prompt the user to reproduce. There are two ways of guidance: complete path and step-by-step guidance. The complete guidance provides the user with a one-time reproduction path. If the user seeks further help, detailed step-by-step guidance will be provided, and the guidance path will be continuously updated as the user's status jumps until the single guidance ends.

[0011] The present invention is characterized in that:

[0012] 1. For the first time in the testing field, a conversational robot was introduced to assist users in reproducing software defects.

[0013] 2. For the first time, traditional computer technology is used to analyze logs and apply them to guidance. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a flow chart of the operation of the system of the present invention.

[0015] Figure 2 This is the flow chart of the model conversion module.

[0016] Figure 3 To reproduce the boot flow chart.

[0017] Figure 4 This is a screenshot of the interface of the present invention. DETAILED DESCRIPTION

[0018] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification.

[0019] This patent implements automatic generation and reproduction guidance of defect reproduction paths based on log analysis and dialogue systems. It mainly uses image understanding technology. The specific key technologies involved include deep convolutional neural network (CNN), OCR technology, Canny technology, AI dialogue robot technology, etc.

[0020] 1. Test log data preprocessing

[0021] In the present invention, we perform model conversion based on the test product to assist the user's reproduction process. In order to achieve the goal, the test product data needs to be standardized first. We convert the test log into a sequence similar to a linked list, where the nodes are page screenshots or exception information during the test, and the links between the nodes are the test steps. Among them, the steps include step categories (click, scroll, input text, etc.), step coordinates, and step additional information such as input strings. Thereby, the original test log is standardized to facilitate subsequent model generation.

[0022] 2. Page identification and merging

[0023] In the present invention, we determine whether they are the same page based on the similarity between GUI screenshots. Among them, the Canny technology in computer vision is used to segment the GUI screenshots, identify the positions of each control, and then extract the structural information of the image based on the arrangement of the controls. Deep learning technology is used to find the embedding vector of each control, and the vector is used to represent the content information of each control. The similarity between pages is comprehensively obtained based on the structural information and content information, and similar pages are merged. Pop-up windows are identified as special pages.

[0024] 3. Establish a page jump model

[0025] In the present invention, we use an undirected multi-edge graph structure as the basic data structure of the page jump model, where the page and defect information are used as the nodes of the graph, and the test steps are used as the edges of the graph. For steps that do not cause page jumps, they are added as edges with the same starting point and end point, but are given a lower weight when guiding the subsequent selection steps.

[0026] 4. Build a dialogue system

[0027] In the present invention, a dialogue robot with Rasa as the core is used to build a complete dialogue system to interact with users and provide guidance for users. The dialogue robot includes complete intention recognition required for guidance, can recognize the user's guidance path request, guidance step inquiry, guidance result notification and other intentions, and cooperate with the guidance system to complete the guidance function.

[0028] 5. Conduct user monitoring

[0029] In the present invention, the system's own event capturer is used to analyze the user's current operation, and the current page is screenshotted and put into the model for comparison. Through user monitoring, the current user's operation can be obtained, the user's location can be determined, and a basis for guidance can be provided.

[0030] 6. Guide strategy selection

[0031] In the present invention, we combine multiple guidance strategies to improve the quality of guidance provided to users. In the selection of the guidance path, the shortest path is preferred as the simplest method, and the user is guided to reproduce according to the shortest path from the initial page to the terminal page. When the shortest path fails, the path of the last ten steps and the path with the longest LCS sequence of the last 10 steps are selected in turn for guidance. When the user deviates from the guidance path, the nearest return path is found again and attached to the original path for guidance.

[0032] The test reproduction guidance method based on test log analysis and human-computer interactive dialogue of the present invention mainly includes:

[0033] Model conversion module: Utilize the test result products and convert them into page jump models according to structural similarity and content similarity for subsequent guidance.

[0034] Detection module: obtains the user's operation and the user's current page. Determines the user's position in the generated model, determines whether the user's operation is correct, and cooperates with the guidance module to give prompts.

[0035] Guidance module: Assists users and guides them to reproduce defects in test results according to specific strategies. Generates guidance paths in real time and informs users through conversational robots.

[0036] Conversational robot module: conveys the guidance information of the guidance module to the user in a user-friendly manner, and passes the user's intention to the guidance module.

[0037] In this example, the test results come from the automated testing of a real APP. Based on the test log, the page screenshots are merged and the step data is added to form a page jump model. According to the user's intention to start guidance expressed to the conversational robot, the generated jump model is loaded into the guidance module, and the real-time monitoring of user operations is started to start guidance. The conversational robot returns the click space provided by the guidance module, predicts the page that will appear, and asks the user for the guidance result when the guidance is predicted to be completed, thereby realizing a complete guidance process.

Claims

1. A test reproduction guidance method based on test log analysis and human-computer interactive dialogue, characterized in that The model built based on the automated test logs generated in advance provides users with guidance on how to reproduce software defects in an interactive and friendly way when they need it. The steps of this method are as follows: 1) The user provides test logs and additional products (including automated test steps, screenshots, etc.) and uses them to build a model. 2) The model conversion module in this method analyzes the test result product and constructs a GUI jump model. The model conversion module identifies the screenshot as a specific GUI page, merges duplicate pages, and finally establishes a jump relationship between pages to construct a jump model. 3) The user requests defect reproduction guidance, and the system starts to guide the reproduction of the defect. This method monitors the user's behavior and provides guidance to the user at the appropriate time. This part mainly includes: 3.1) Monitoring module: monitors and records the user's operations to determine the user's position on the jump model, and prompts the user through the dialogue module when the user deviates. 3.2) Guidance module: Generates a guidance path for the user based on the position of the user's current page on the jump model. 3.3) Conversational robot module: Receives user instructions and passes them to the guidance module in a user-friendly manner, and presents the results of the guidance module to the user.

2. According to claim 1, a method for automated test reproduction guidance based on test result product analysis, heuristic methods and dialogue systems is characterized in that: in the model conversion module, GUI structure matching and feature vectors extracted by deep convolutional neural networks are used to calculate the similarity between each page, and pages with higher similarity are merged.

3. The method of automated test reproduction guidance based on test result product analysis, heuristic method and dialogue system according to claim 1, characterized in that: In the monitoring module, the user's operations are obtained and recorded, and the user's state jump is determined based on the similarity of the pages.

4. The method of automated test reproduction guidance based on test result product analysis, heuristic method and dialogue system according to claim 1, characterized in that: In the guidance module, according to the position of the current page in the jump model, three heuristic strategies are used to generate a guidance path for users in descending order of priority, including: 4.1) Take the shortest path from the initial page to the abnormal page as the target path. 4.2) Take the last 10 steps leading to the anomaly as the target path. 4.3) Take the last 10 steps to the exception as the original path, obtain all complete paths through depth-first search, and take the path with the longest common subsequence with the original path as the target path.

5. The method of automated test reproduction guidance based on test result product analysis, heuristic method and dialogue system according to claim 1, characterized in that: In the conversational robot module, the intent recognition and dialogue management provided by the open source framework Rasa are used to receive user instructions and call the guidance module based on the context.