A method for mobile application crowd testing report clustering based on deep image semantic mining

Abstract

A method for clustering crowdsourced test reports for mobile applications based on deep image semantic mining includes a feature extraction module, a distance calculation module, a constraint rule construction module, and a semi-supervised clustering module. In the feature extraction module, four features—structural features, content features, defective behaviors, and reproduction steps—are extracted to represent the crowdsourced test reports composed of screenshots and text descriptions. In the distance calculation module, using the extracted features, different distance algorithms are employed to calculate the semi-supervised clustering distance for each pair of test reports based on their four features. A weighted algorithm is then used to obtain the overall semi-supervised clustering distance between each pair of test reports. In the constraint rule construction module, semantic constraint rules, including Must-Link and Cannot-Link, are constructed. In the semi-supervised clustering module, guided by the semantic constraint rules, the K-Medoids algorithm is used to perform semi-supervised clustering of the test reports.

Description

Technical Field

[0001] This invention belongs to the field of software testing, specifically relating to test report clustering. After in-depth analysis of the provided test reports using screenshots and text descriptions, reasonable clustering results are obtained through semantic binding rules. Background Technology

[0002] Crowdsourcing is a distributed problem-solving and production organization model that has been widely applied in fields such as artificial intelligence, natural language processing, and computer vision. Crowdsourced testing is an emerging software testing method that dominates mobile application testing. The common workflow of crowdsourced testing is as follows:

[0003] Requesters post testing tasks on crowdsourced testing platforms, and then crowdsourced workers perform these tasks and submit test reports to the platform. Most mainstream crowdsourced testing platforms process the test reports and provide feedback to the requesters.

[0004] On most mainstream crowdsourced testing platforms, crowdsourced projects are required to submit reports describing errors that occurred during the testing process. Each crowdsourced test report should consist of an application screenshot and a text description. The text description includes steps to reproduce the test event that triggered the defect and a description of the defect focusing on it. The application screenshot and text description help application developers better review the report.

[0005] However, for mobile app developers, exhaustively testing mobile applications across all runtime environments is a daunting task. Crowdsourced testing addresses this shortcoming due to its openness. In crowdsourced testing, testing tasks are distributed among a large number of crowdsourced workers, utilizing their diverse testing environments to achieve the testing objectives. However, the openness of crowdsourced testing also introduces risks to mobile app testing. A significant problem is duplicate reporting. Most reports reflect similar defects, hindering mobile app developers from efficiently reviewing reports and addressing defects.

[0006] Clustering methods are widely used to address the problem of duplicate test reports, employing unsupervised approaches and feature extraction from crowdsourced test reports. Most existing work focuses solely on the textual information of the test reports to cluster duplicate reports. Recently, crowdsourced test reports have incorporated mobile application screenshots and text descriptions. Mobile application screenshots are powerful carriers of defect information, but have often been underestimated in previous research. Properly utilizing the semantic connections between screenshots and text descriptions can better characterize defects.

[0007] Based on the above work, this invention proposes a method for clustering crowdsourced test reports of mobile applications based on deep image semantic mining. It can extract semantic features from screenshots and text descriptions to characterize crowdsourced test reports, deeply explore the semantic relationship between screenshots and text descriptions, form semantic constraint rules to guide the clustering process, and complete the clustering of crowdsourced test reports under the guidance of semantic constraint rules. Summary of the Invention

[0008] The problem this invention aims to solve is that, due to the open nature of the crowdsourced testing paradigm, crowdsourced workers submit a large number of inconsistent and duplicate test reports, which hinders developers from effectively reviewing reports and identifying errors. Our invention can deeply mine the image information from screenshots of crowdsourced test reports and the semantic information from text descriptions to form semantic constraint rules that guide the clustering process, thereby clustering crowdsourced test reports and solving the problem of duplicate test reports.

[0009] The technical solution of this invention is: a method for clustering mobile application crowdsourced test reports based on deep image semantic mining. Its characteristic is the ability to extract features from application screenshots and text descriptions provided in the crowdsourced test reports, form semantic constraint rules based on these features, and complete clustering under the guidance of these semantic constraint rules. This clustering method comprises the following four modules:

[0010] 1) Feature extraction module: Performs feature extraction on screenshots and text descriptions, specifically extracting structural and content features from screenshots and extracting defective behaviors and reproduction steps from text descriptions. When extracting structural features, a novel method combining traditional computer vision algorithms and deep learning techniques is employed to process screenshots. All valid components are extracted from the screenshot and organized into a four-level tree structure. Since controls may be extracted from blank areas, this patent further employs a CNN model and VGG-16, which are effective for image classification, to identify the type of each candidate control. When extracting content features, since the screenshot has already been characterized to construct structural features, all leaf nodes of the screenshot structure, i.e., component screenshots, are taken to represent the content features of the screenshot. When extracting defective behaviors, the widely used Word2Vec model is used to extract defective behaviors. To make the Word2Vec model more suitable for mobile application crowdsourcing testing scenarios, this patent uses a domain-specific keyword list to identify synonyms, antonyms, and polysemous words in mobile application testing. When extracting reproducible steps, a component parser algorithm is used to analyze the part-of-speech tags of text segments, extracting user operations and operation objects to form the operation sequence of the reproducible steps.

[0011] 2) Distance Calculation Module: Utilizing the extracted features, different distance algorithms are used to calculate the semi-supervised clustering distance for each pair of test reports based on their four features. A weighted algorithm is then used to derive the overall semi-supervised clustering distance between each pair of test reports. Structural features are in the form of tree structures, calculated using the advanced tree edit distance algorithm APTED. Content features are in the form of image vectors; traditionally, the distance to the entire app screenshot is directly calculated, but this patent considers component granularity here. Defective behaviors are in the form of text vectors, and Euclidean distance is typically used in pairs to represent the distance to defective behaviors. Reproduction steps are in the form of a sequence of operation nodes; therefore, to incorporate semantics into the distance calculation, this patent does not directly convert the text to vectors. Instead, the reproduction steps are treated as a sequence, calculating the text similarity between all nodes, and then employing the Dynamic Time Warping (DTW) algorithm.

[0012] 3) Constraint Rule Construction Module: This patent uses two constraints to guide the clustering process. Both constraints take into account screenshots, text descriptions, and semantic relationships between them, including Must-Link and Cannot-Link: Must-Link means that two test reports must be clustered into the same class if they satisfy the Must-Link rule; Cannot-Link means that two test reports cannot be clustered into the same class if they satisfy the Cannot-Link rule. Must-Link and Cannot-Link rules are transitive.

[0013] 4) Semi-supervised clustering module: Guided by semantic constraint rules, semi-supervised clustering is performed on the test reports using the K-Medoids algorithm. The specific process involves first determining the k-value (target number of clusters) using the elbow method; then selecting initial cluster centroids by randomly selecting k test reports as centroids. However, these k reports may have must-links. This patent eliminates this possibility based on a pre-constructed must-link dictionary. By continuously reselecting k test reports and verifying semantic binding rules, k centroids without must-links are selected. Next, the semi-supervised distances are calculated between the test reports (excluding the centroids) and the corresponding test reports of each cluster centroid. Finally, the calculation results are corrected based on semantic constraint rules. For the target test report, if multiple clusters satisfy the existing rules... If a test report that meets the Must-Link criteria does not contain any test reports that meet the Cannot-Link criteria, then that test report will be clustered into the cluster with the smallest semi-supervised distance to the cluster center. If multiple clusters exist that meet the criteria of having both test reports that meet the Cannot-Link criteria and not containing any test reports that meet the Must-Link criteria, then that test report will be clustered into the cluster that neither contains any test reports that meet the Cannot-Link criteria nor contains any test reports that meet the criteria of Cannot-Link criteria, and also contains any test reports that contain any test reports that meet the criteria of Cannot-Link criteria. Otherwise, the test report is considered to be in conflict, and the target test report remains in its original cluster. Finally, the center points of each cluster are updated, and the process of calculating the semi-supervised distance and updating the center points is continuously iterated to obtain the semi-supervised clustering results.

[0014] The features of this invention are:

[0015] 1. A novel and effective representation and representation feature is proposed for the first time for application screenshots and text descriptions in crowdsourced test reports.

[0016] 2. For the first time, semantic constraint rules are introduced to guide semi-supervised clustering algorithms.

[0017] 3. A novel method for clustering crowdsourced test reports using depth images and semantics is proposed for the first time. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the implementation of the present invention.

[0019] Figure 2 Here is a detailed flowchart of the feature extraction module.

[0020] Figure 3 This is a detailed flowchart of the distance calculation module.

[0021] Figure 4 A detailed flowchart for building the constraint rules module.

[0022] Figure 5 A detailed flowchart for the semi-supervised clustering module. Detailed Implementation

[0023] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification.

[0024] This patent implements crowdsourced test report clustering, mainly employing image understanding technology, text analysis technology, and distance calculation technology. The specific key technologies involved include traditional computer vision (CV) algorithms, deep learning (DL) technology, deep convolutional neural networks (CNN), VGG-16 model technology, Word2Vec model technology, tree edit distance algorithm (APTED), and K-Medoids algorithm.

[0025] 1. Image Feature Extraction

[0026] In this invention, we employ a novel method combining traditional computer vision algorithms and deep learning techniques to process screenshots. This method extracts structural features from screenshots provided in crowdsourced test reports, transforming an image into a four-level tree structure. Traditional computer vision algorithms utilize mature CV techniques for object detection, such as feature descriptors. Before the rise of deep learning, image feature extraction often involved multiple CV algorithms. Traditional computer vision algorithms have several drawbacks; for example, feature extraction becomes increasingly difficult as the number of feature categories in an image increases. Deep learning introduces the concept of end-to-end learning. Deep learning models are trained on given data, where neural networks discover underlying patterns within image categories and automatically extract the most descriptive and salient features for the target category. The combination of traditional computer vision algorithms and deep learning techniques enables better completion of image feature extraction tasks.

[0027] 2. Control type identification

[0028] In this invention, we employ a Convolutional Neural Network (CNN) model for image classification and a Visual Geometric Group Network (VGG-16) to identify the type of each candidate control, improving the efficiency of extracting existing application controls. A CNN is a type of feedforward neural network that includes convolutional computation and has a deep structure. CNNs possess representation learning capabilities, enabling translation-invariant classification of input information according to their hierarchical structure. CNNs are constructed by mimicking biological visual perception mechanisms, allowing for both supervised and unsupervised learning. The shared kernel parameters within their hidden layers and the sparsity of inter-layer connections enable CNNs to handle gridded features with relatively low computational cost. The Visual Geometric Group Network demonstrates that increasing the network depth can significantly impact the final performance of the network. The convolutional layers and pooling layers in the Visual Geometric Group Network all use the same kernel parameters, effectively using small-sized convolutional kernels to achieve the equivalent of large-sized kernels.

[0029] 3. Generate a keyword database

[0030] In this invention, words in the test report are mapped to vectors to determine their frequency of occurrence, thereby generating a defect behavior keyword library. To make the Word2Vec model more suitable for mobile application crowdsourcing testing scenarios, this invention uses a domain-specific keyword list to identify synonyms, antonyms, and polysemous words in mobile application testing. Word2Vec technology is a shallow, two-layer neural network that generates word vectors. After training, it can reconstruct word text. After training, Word2Vec can map each word to a vector.

[0031] 4. Feature distance calculation

[0032] In this invention, we employ the advanced tree edit distance algorithm APTED to calculate the feature distance between structural features of each pair of test reports. Tree edit distance refers to the minimum number of operations required to transform one tree structure into another. The tree edit distance algorithm considers three operations: deletion, insertion, and renaming. The algorithm first uses a single-path function to calculate the edit distance between two trees, then recursively calculates the edit distance between a subtree of one tree and all subtrees of another tree, thereby sorting the distances.

[0033] 5. Test report clustering

[0034] In this invention, we employ the K-Medoids algorithm to perform semi-supervised clustering of test reports under the guidance of semantic constraint rules. The K-Medoids algorithm is a clustering method based on "representative objects." Instead of using the average value, the K-Medoids algorithm uses the object at the center of the cluster, i.e., the medoid, as the reference point. The K-Medoids algorithm is suitable for small samples and exhibits good robustness to noise.

[0035] The method for clustering mobile application crowdsourced test reports based on deep image semantic mining of the present invention mainly includes:

[0036] Feature extraction module: Extracts four features—structural features, content features, defective behavior, and reproduction steps—to represent the crowdsourced test report composed of screenshots and text descriptions.

[0037] Distance calculation module: Using the extracted features, different distance algorithms are used to calculate the semi-supervised clustering distance of the four features of each pair of test reports, and a weighted algorithm is used to obtain the overall semi-supervised clustering distance between each pair of test reports.

[0038] Constraint rule building module: Builds semantic constraint rules, including Must-Link and Cannot-Link.

[0039] Semi-supervised clustering module: Guided by semantic constraint rules, the K-Medoids algorithm is used to perform semi-supervised clustering on the test reports.

[0040] In this example, we collected 847 crowdsourced test reports from 18 mobile applications covering different categories, and invited three mobile application testing experts to manually annotate the defects presented in the test reports. After feature extraction and distance calculation of these test reports, we constructed semantic constraint rules using Must-Link and Cannot-Link constraints, and performed semi-supervised clustering of the test reports under the guidance of the semantic constraint rules. Finally, we compared the clustering results with the manually annotated results and found that this invention can effectively handle the problem of duplicate test reports.

Claims

1. A method for clustering crowdsourced test reports of mobile applications based on deep image semantic mining, characterized by combining screenshots and text descriptions in the crowdsourced test reports to deeply mine image semantic information, and introducing semantic constraint rules to perform semi-supervised clustering of the crowdsourced test reports. The steps of this method are as follows: 1) Feature extraction module: This module uses four features—structural features, content features, defective behavior, and reproduction steps—to represent the crowdsourced test report, which consists of screenshots and text descriptions. It mainly includes: 1.1) Structural Features: The structural features extract the graphical user interface structure describing the application screenshot. Candidate application controls are extracted from the application screenshot using computer vision algorithms. A CNN model and VGG-16 are used to identify the type of each candidate application control. Areas in the screenshot containing only image resources are removed, and the application controls are obtained and organized into a four-level tree structure. 1.2) Content features, extracted from the screenshot, describe the graphical user interface component content of the application screenshot, including image controls and button controls, and the content features are represented by the component screenshots corresponding to the leaf nodes of the screenshot structure; 1.3) Defective behaviors: Defective behaviors are extracted from text descriptions, describing defective situations and closely related to the testing expectations of crowdsourcing workers. The extraction of defective behaviors is achieved by adopting the widely used Word2Vec model. 1.4) Reproduction steps: The reproduction steps are extracted from the text description, which describes the user's operation sequence from application startup to defect occurrence. By using the component parser algorithm to analyze the part-of-speech tags of the text segments, the user's operations and operation objects are extracted to form the operation sequence of the reproduction steps. 2) The distance calculation module uses the extracted structural features, content features, defect behaviors and reproduction steps to calculate the semi-supervised clustering distance of each pair of test reports on the four features using the corresponding distance algorithm, and uses a weighted algorithm to obtain the overall semi-supervised clustering distance between each pair of test reports. 3) Constraint rule construction module: Constructs semantic constraint rules, including Must-Link rules and Cannot-Link rules. The Must-Link rule indicates that two test reports will cluster into the same cluster if they satisfy the Must-Link rule, while the Cannot-Link rule indicates that two test reports will not cluster into the same cluster if they satisfy the Cannot-Link rule. 4) Semi-supervised clustering module: Guided by semantic constraint rules, semi-supervised clustering is performed on the test reports. The clustering method used is the K-Medoids algorithm, and the process is as follows: 4.1) The elbow method is used to determine the k value of the clustering algorithm, i.e., the target number of clusters; 4.2) Select the initial center point of the cluster; 4.3) Except for the center point, the semi-supervised distance between the test reports and the test reports corresponding to the center points of each cluster is calculated in turn; 4.4) Correct the calculation results based on semantic constraint rules; 4.5) Update the center point of each cluster; 4.6) Iterate from 4.3 to 4.5 to obtain the semi-supervised clustering results; 2. The method for clustering mobile application crowdsourced test reports based on deep image semantic mining according to claim 1, characterized in that: In the feature extraction module, when extracting structural features, computer vision algorithms are used to extract application controls from application screenshots, and CNN models and VGG-16 are used to identify the type of each candidate application control in order to remove areas in the screenshot that only contain image resources. When extracting defective behaviors, a keyword list for specific domains is used to identify synonyms, antonyms, and polysemous words in mobile application testing, making the Word2Vec model applicable to mobile application crowdsourcing testing scenarios.

3. The method for clustering mobile application crowdsourced test reports based on deep image semantic mining according to claim 1, characterized in that: The content features are in the form of image vectors, and the distances corresponding to the content features are calculated based on the component granularity; the reproduction steps are in the form of an operation node sequence, the reproduction steps are used as an operation node sequence, the text similarity between operation nodes is calculated, and the distances corresponding to the reproduction steps are calculated using a dynamic time warping algorithm.

4. The method for clustering mobile application crowdsourced test reports based on deep image semantic mining according to claim 1, characterized in that: The constraint rule construction module uses Must-Link and Cannot-Link rules to guide the clustering process. Both Must-Link and Cannot-Link rules are constructed based on screenshots, text descriptions, and the semantic relationship between screenshots and text descriptions. Furthermore, Must-Link and Cannot-Link rules are transitive.

5. The method for clustering mobile application crowdsourced test reports based on deep image semantic mining according to claim 1, characterized in that: In the semi-supervised clustering module, when selecting the initial cluster centroids, k test reports are randomly selected as centroids, and the k test reports are verified according to the constructed Must-Link dictionary. When there is a Must-Link relationship between the k test reports, k test reports are reselected and verified again until k centroids without Must-Link relationships are selected. When correcting the calculation results based on semantic constraint rules, for a target test report, if there are multiple clusters that satisfy both Must-Link rules and Cannot-Link rules, the target test report is clustered into the cluster with the smallest semi-supervised distance to the cluster centroid. If multiple clusters exist that contain test reports that conform to the Cannot-Link rule but not test reports that conform to the Must-Link rule, then the target test report is clustered into the cluster that does not contain test reports that conform to the Cannot-Link rule and has the smallest semi-supervised distance to the center point; otherwise, it is considered a conflict, and the target test report remains in its original cluster.

Images