A method for building a knowledge graph-based mobile application function point test knowledge base

Abstract

A method for building a knowledge graph-based mobile application function point test knowledge base, comprising a knowledge graph feature extraction module, a knowledge graph relationship extraction module, a knowledge graph co-reference resolution module and a knowledge graph intelligent query module. The knowledge graph feature extraction module is mainly responsible for effectively extracting and disassembling a large number of mobile application crowd testing reports with repeated steps to help build a knowledge graph. The knowledge graph relationship extraction module is to integrate the data analyzed in the knowledge graph feature extraction module. The knowledge graph co-reference resolution module is to upload the data sorted by the knowledge graph feature extraction module and the knowledge graph relationship extraction module to the designated Neo4j graph database. The knowledge graph intelligent query module is to help users quickly understand the knowledge graph and provide guidance for mobile application automated testers on what to do next in automated function point testing.

Description

Technical Field

[0001] This invention belongs to the field of software testing, specifically generating a knowledge graph based on crowdsourced test reports. It involves extracting text and image features from the provided crowdsourced test reports, constructing knowledge graph relationships, resolving coreference, and finally generating the knowledge graph. Background Technology

[0002] In recent years, the mobile internet and mobile devices have developed rapidly, and mobile applications have become intertwined with all aspects of social life. Therefore, developers are increasingly focusing on testing techniques to ensure the quality of mobile applications. Mobile applications are characterized by diverse versions and rapid iterations, making traditional software testing techniques insufficient. Consequently, new testing technologies for mobile applications have become a major research topic in both academia and industry. Currently, the main automated testing technologies are Monkey and Appium; the former performs random click testing, while the latter requires users to write test scripts for execution.

[0003] While current mobile application testing tools can significantly reduce the cost of manual testing, they still have major drawbacks. For example, the Monkey framework, while applicable to all mobile applications, relies on random clicks, lacking logical consistency in its testing and failing to reproduce logical vulnerabilities that require a specific process. In contrast, Appium uses manually written test scripts, which are logically sound, but these scripts have poor portability, making them unusable for the same mobile application across different platforms. Furthermore, we can observe that the execution flow of certain scripts is highly similar across different mobile applications, differing only in application version, layout, and resolution. This necessitates writing multiple repetitive automated test scripts, significantly increasing labor costs and slowing down the testing process.

[0004] We've found that the main reason for the redundant manpower costs in current mainstream mobile application testing tools is the lack of a holistic understanding of mobile application characteristics. Even minor pixel changes or layout tweaks necessitate rewriting test scripts. If we could automate the identification of these subtle differences, we could apply functional test scripts to all mobile application products. Furthermore, we've discovered that many mobile applications share similar operational flows, such as login and ticket purchase. We term these general macro-level logics, like login and ticket purchase, as the functional points of a mobile application. If we can build relevant test libraries based on these functional points, it can help Appium automate testing for all mobile applications.

[0005] Therefore, we propose a method for building a mobile application feature testing knowledge base based on knowledge graphs. By providing compliant crowdsourced test reports, we extract knowledge graph features using text part-of-speech analysis and screenshot control recognition. We then use CNN controls to identify and extract macro-level entity concepts to derive knowledge graph relationships. Finally, we construct relationships between entities through coreference resolution, building a powerful mobile application automated testing knowledge base to help existing mobile application automated testing processes test mobile applications more efficiently and conveniently. Summary of the Invention

[0006] The problem this invention aims to solve is the redundancy in manpower costs of mainstream mobile application testing tools, which require writing redundant test scripts for different similar applications and functionalities. Our invention can extract knowledge graph features and relationships from reported test reports in mobile applications, utilize coreference resolution to construct relationships between entities, and generate a mobile application automated testing knowledge base to assist in existing mobile application automated testing.

[0007] The technical solution of this invention is: a method for building a mobile application functional point testing knowledge base based on knowledge graphs, characterized by its ability to generate a powerful automated testing knowledge base for mobile applications based on compliant crowdsourced testing reports. This generation method comprises the following four modules:

[0008] 1) Knowledge Graph Feature Extraction Module: This module is primarily responsible for effectively extracting and decomposing information from a large number of mobile application crowdsourcing test reports with reproducible steps to help build a knowledge graph. Feature extraction in this part uses Optical Character Recognition (OCR) technology, supporting images containing Chinese, English, or mixed text, extracting all text information from the images, and providing information such as content and location. For screenshots from crowdsourcing tests, we use the Canny algorithm to extract key controls through edge recognition. The results of the first two methods are then analyzed using the Merge method to extract key information points from the images. Next, the text content is analyzed using Jieba, and finally, both the image and text analyses are saved.

[0009] 2) Knowledge Graph Relationship Extraction Module: This module integrates the data analyzed in the knowledge graph feature extraction module. While the feature extraction module extracts features from text and images using various algorithms, it doesn't connect the content of the two. This module integrates the text and image components. The relationship extraction process first identifies text in screenshots similar to the macroscopic concept within the image, using this as OCR information. It then finds all other similar OCRs, locates the text's position based on the OCR information, and identifies the corresponding component information. Finally, it uses a CNN to identify the component type and determine its class. A node class is then created, storing the operation, logic, OCR, component, and component category information, arranged sequentially according to the operation order, and finally written to a file and saved to a specified folder.

[0010] 3) Knowledge Graph Coreference Resolution Module: This module uploads the data processed by the knowledge graph feature extraction and relationship extraction modules to the designated Neo4j graph database. During the generation or expansion of the knowledge graph, newly added nodes are compared and analyzed with existing nodes to resolve coreference. During construction, the system adds the successor order of nodes to the graph, identifies identical nodes to eliminate redundancy, establishes connections between similar nodes (referred to as similarity), and marks the head and tail nodes in each functional scenario, preserving the acquired knowledge information in the knowledge graph to the greatest extent possible.

[0011] 4) Knowledge Graph Intelligent Query Module: This module helps users quickly understand knowledge graphs and provides guidance for mobile application automation testers on the next steps for automated feature testing. Knowledge graph queries have two modes: a global search mode, which traverses all nodes in the graph, offering a wide scope but poor timeliness; and a head search mode, which uses a given feature to narrow down the comparison range and speed up the search.

[0012] The features of this invention are:

[0013] 1. This paper is the first in the field of mobile application testing to propose using knowledge graphs to build a knowledge base for testing mobile application function points.

[0014] 2. This is the first time that deep learning and computer vision technologies have been applied to knowledge graph construction.

[0015] 3. For the first time, the knowledge base for testing application functionalities is automated, and the knowledge base is updated within 3 seconds for each compliant crowdsourced test report. Attached Figure Description

[0016] Figure 1This is a flowchart illustrating the implementation of the present invention.

[0017] Figure 2 This is a detailed flowchart of the knowledge graph feature extraction module.

[0018] Figure 3 This is a detailed flowchart of the knowledge graph relationship extraction module.

[0019] Figure 4 This is a detailed flowchart of the knowledge graph coreference resolution module.

[0020] Figure 5 This is a detailed flowchart of the knowledge graph intelligent query module. Detailed Implementation

[0021] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification.

[0022] This patent describes a method for building a knowledge base for testing the functional points of mobile applications based on knowledge graphs. It mainly employs image understanding and text analysis technologies, and the specific key technologies involved include deep convolutional neural networks (CNN), OCR technology, Canny technology, word2vec technology, synonyms technology, and jieba technology.

[0023] 1. Image Feature Extraction

[0024] In this invention, we use Canny and OCR technologies to extract image features. First, we use OCR to extract all text information from the screenshot, including text content and location. Then, we use the Canny edge extraction algorithm to extract the edges of the screenshot components, segmenting them into smaller parts. Our Merge algorithm then merges the results of the Canny and OCR algorithms, primarily focusing on merging containment and intersection relationships. Finally, we use SIFT technology to extract feature points from the image to improve the accuracy of the matching tool.

[0025] 2. Text Feature Extraction

[0026] In this invention, we employ word2Vec and jieba techniques for text feature extraction. The Word2Vec algorithm is a shallow neural network. Compared to previous word vector analysis models, Word2Vec performs better because it considers the context of the text. Simultaneously, its reduced dimensionality significantly improves training speed. Furthermore, Word2Vec is highly versatile and can be used in various NLP research projects. The Jieba technique efficiently scans the word graph based on a prefix dictionary, generating a directed acyclic graph (DAG) containing all possible words in a sentence, and producing a corresponding word segmentation table. We then use word2Vec to extract word vectors and jieba to extract word segments, constructing the text's features.

[0027] 3. Control recognition

[0028] In this invention, we employ a Convolutional Neural Network (CNN) to perform control recognition on extracted component screenshots, transforming an image into a feature vector, and then using the neural network to determine the component category. A CNN is a type of feedforward neural network that includes convolutional computation and has a deep structure. CNNs possess representation learning capabilities, enabling translation-invariant classification of input information according to their hierarchical structure. CNNs are constructed by mimicking biological visual perception mechanisms, allowing for both supervised and unsupervised learning. The shared parameters of convolutional kernels within their hidden layers and the sparsity of inter-layer connections enable CNNs to perform gridded features with relatively low computational cost. Using CNN technology, we can perform category analysis on the extracted components, labeling them as buttons, input boxes, etc., thereby improving data accuracy.

[0029] 4. Spatial Isolation Analysis

[0030] In this invention, we use the Canny technique to segment application screenshots and extract each individual control in the image for analysis. The Canny edge detection algorithm first applies Gaussian filtering to smooth the image and remove noise; secondly, it finds the intensity gradient of the image; then, it applies non-maximum suppression to eliminate false edge detections; next, it applies a double thresholding method to determine possible boundaries; and finally, it uses hysteresis techniques to track the boundaries.

[0031] 5. Chinese Synonym Matching

[0032] In this invention, we use synonyms for synonym matching. Synonyms is a highly efficient and accurate Chinese synonym database, ranking among the best in the field. This database is constructed using the Word2Vec algorithm, trained on Chinese vocabulary from all text within Wikipedia. Using this technology, we can perform tasks such as synonym searching and word / sentence similarity analysis.

[0033] 6. Coreference resolution

[0034] In this invention, we utilize coreference resolution to construct a graph database. For each functional scenario dimension file, the system uploads the nodes recorded sequentially in the file. During the upload process, the system adds the successor-successor order between nodes to the graph, while also searching for identical nodes to eliminate redundancy, establishing connections between similar nodes (referred to as similarity), and marking the head and tail nodes in each functional scenario, thus preserving the acquired knowledge information in the knowledge graph to the greatest extent possible.

[0035] The method for building a knowledge base for testing mobile application functionalities based on knowledge graphs, as described in this invention, mainly includes:

[0036] Knowledge Graph Feature Extraction Module: This module is mainly responsible for effectively extracting and decomposing information from a large number of mobile application crowdsourcing test reports with reproducible steps to help build a knowledge graph.

[0037] Knowledge Graph Relationship Extraction Module: This module integrates the data analyzed in the Knowledge Graph Feature Extraction Module. The Knowledge Graph Feature Extraction Module only extracts features from text and images using various algorithms, but it does not link the content of the two together. This module can integrate the text and image parts together.

[0038] Knowledge Graph Coreference Resolution Module: This module is used to upload the data processed by the Knowledge Graph Feature Extraction Module and the Knowledge Graph Relationship Extraction Module to the specified Neo4j graph database.

[0039] Knowledge Graph Intelligent Query Module: This module is designed to help users quickly understand knowledge graphs and provide guidance to mobile application automation testers on what to do next in automated feature testing.

Claims

1. A method for building a knowledge base for testing functional points of mobile applications based on knowledge graphs, characterized in that: 1) Users upload a compliant crowdsourcing test report; 2) Perform feature extraction and graph construction on application screenshots uploaded by users, specifically including: 2.1) Knowledge Graph Feature Extraction: Effective information extraction and decomposition of numerous mobile application crowdsourcing test reports with reproducible steps are performed to aid in the construction of a knowledge graph. Various algorithms are used to extract features from text and images. Canny is used to identify control edges and extract control information. OCR technology is used to extract text and location information. SIFT technology is used to extract feature points from images. CNN technology is used for control category recognition. The results of these technologies are combined to generate image features. For text features, jieba and word2Vec technologies are used for word segmentation and word vector extraction to construct text features. 2.2) Knowledge Graph Relationship Extraction: The data analyzed in 2.1) is integrated, specifically including: First, finding text in screenshots similar to macro concepts from the text in the images as OCR information, and finding all other OCRs similar to the OCR. Second, finding the location of the text based on the OCR information and then finding the corresponding component information. Third, using CNN to identify the type of the component to determine the component class. Finally, creating node classes to store the operation, logic, OCR, component, and component category information, and arranging them in the order of operation, and finally writing them into a file and saving them to the specified folder. 2.3) Knowledge Graph Coreference Resolution: The data prepared in 2.1) and 2.2) is uploaded to the specified Neo4j graph database. During the generation or expansion of the knowledge graph, the synonyms technology is used to compare and analyze the newly added nodes with the original nodes in the knowledge graph to complete the coreference resolution operation. During the construction process, the system adds the order between nodes to the graph, and at the same time, it finds the same nodes to eliminate redundancy, finds similar nodes to establish connections, and marks the head node and tail node in each functional scenario, so as to retain the acquired knowledge information in the knowledge graph to the greatest extent. 2.4) Intelligent Knowledge Graph Query: Helps users quickly understand knowledge graphs and provides guidance for mobile application automation testers on subsequent operations of automated function point testing; utilizes two query methods to provide testing guidance to users; One approach is global search, which traverses and queries all nodes in the knowledge graph, covering a wide range but with slow query speed. The other approach is head search, which uses a given feature to help the knowledge graph narrow down the comparison range and speed up the search.

Images