A knowledge-enhanced large model driven crowdsourcing test report prioritization method

Abstract

The application discloses a kind of knowledge enhanced large model driven crowd-sourcing test report priority sequencing methods, including clustering deduplication module, knowledge empowerment classification module and intelligent sorting module.Cluster deduplication module utilizes Emb e Adding model converts report text into high-dimensional vector, and representative reports are selected by calculating cosine similarity clustering deduplication and reducing subsequent processing cost.Knowledge empowerment classification module extracts entities, expands knowledge graph and performs TF-IDF matching classification on representative reports to generate accurate classification results and classification evidence.Intelligent sorting module sorts representative reports through Graph RAG, and then expands non-representative reports through cross-pollination strategy, aiming to reduce Token cost, improve sorting diversity and defect discovery capability.

Description

Technical Field

[0001] This invention belongs to the field of software testing. Specifically, it relates to a crowdsourced test report priority ranking method based on Graph RAG and multi-stage clustering. It aims to solve the problems of report redundancy and inefficient ranking in crowdsourced testing scenarios by using multi-stage clustering deduplication, knowledge graph empowerment, and large language model (LLM) intelligent ranking, thereby improving the review efficiency and defect location efficiency of mobile applications in crowdsourced testing scenarios. Background Technology

[0002] With the rapid popularization of the internet and mobile devices, mobile applications have gradually permeated all aspects of social life. Although various applications are updated and iterated frequently, the quality of software varies greatly. Therefore, testing technology for mobile applications has become an important and hot topic in academia and industry.

[0003] Crowdsourced testing, as a large-scale, low-cost testing method, has the advantages of covering multiple devices and usage scenarios, and user-centric data collection. Developers can recruit a large number of testers online, each submitting defect reports including screenshots and text descriptions, thus obtaining rich testing feedback in a short period of time. However, this openness also brings problems such as a large number of reports, information redundancy, and frequent duplicate reports. This not only increases the review burden on developers but also leads to excessively high token costs and insufficient diversity of ranking results during the LLM ranking process, making it difficult to quickly identify key defects, affecting testing efficiency and restricting the widespread application of crowdsourced testing.

[0004] Existing technologies typically rely on simple deduplication based on text feature extraction and similarity calculation, or directly use LLM to sort all reports. These methods fail to fully utilize the structured advantages of knowledge graphs and do not consider the impact of the prevalence of defects on priority. They suffer from problems such as incomplete deduplication, high token costs, and weak targeting in sorting, making it difficult to meet the needs of efficient and accurate report review in crowdsourcing scenarios. With the rapid development of LLM, Embedding technologies, and Knowledge Graphs (KG), new technical approaches have been provided for achieving efficient deduplication, accurate classification, and intelligent sorting.

[0005] Against this backdrop, the present invention combines the technical advantages of Embedding clustering, KG, and LLM. Through the collaborative processing of three core modules, it first achieves report clustering deduplication and representative screening, then performs entity extraction, KG expansion and accurate classification, and finally completes intelligent sorting and sequence expansion. This reduces token costs, improves sorting diversity and defect detection efficiency, and solves the shortcomings of existing technologies. Summary of the Invention

[0006] The present invention addresses the problems of severe report redundancy, insufficient sorting diversity, rigid judgment of defect severity, and low classification accuracy in the analysis of crowdsourced test reports for current mobile applications. Our core idea is to achieve low-redundancy, high-diversity, and high-accuracy prioritization of crowdsourced test reports through the collaborative implementation of three coarse-grained modules: clustering deduplication, knowledge-enabled classification, and intelligent sorting.

[0007] The technical solution of this invention is: a crowdsourcing report priority ranking method based on Graph RAG and multi-stage clustering, characterized by its ability to generate a final priority ranking result through the collaborative processing of three core modules based on the provided text and image reports. This generation method comprises the following three modules:

[0008] 1) Clustering and Deduplication Module: This module takes a large number of natural language test reports provided by the testing platform as input. Its core function is to deduplicate and select representative reports, reducing subsequent processing costs. Specific operations include: using an embedding model to convert the text descriptions of all reports into high-dimensional vectors; calculating the cosine similarity between reports, clustering reports with similarity exceeding a preset threshold into a single cluster, identifying them as different descriptions of the same type of defect; selecting a representative report from each cluster, recording the total number of original reports for each cluster, and performing subsequent high-cost processing steps only on the representative report, while temporarily storing non-representative reports for later processing. This suppresses interference from duplicate reports at the source, achieving both reduced workload and increased diversity.

[0009] 2) Knowledge-Enabled Classification Module: This module processes only representative reports selected by the clustering and deduplication module. Its core functions include entity extraction, knowledge graph expansion, and accurate classification, providing data support and classification criteria for subsequent ranking. Specific operations include: First, using a large language model to extract key entity information from the representative report text, including operations, symptoms, specific pages, and UI elements, and persisting the extraction results to a specified cache directory to form reusable cache files; second, mounting the extracted entities as nodes onto the global knowledge graph, establishing the relationship between reports and entities, constructing a "general-specific" two-layer knowledge graph structure, and enabling multi-hop traversal of the report context; finally, using a classification script to calculate the IDF weights of each entity feature to reflect the discriminative power of each entity, achieving accurate classification of reports and defect types through TF-IDF matching scoring, and extracting the top three high IDF features as classification evidence.

[0010] 3) Intelligent Sorting Module: This module's core functionality involves intelligent sorting of representative reports and cross-expansion of non-representative reports to generate a final highly diverse and low-redundancy sorted sequence. Specific operations include: First, using an LLM (Limited Least Metric) to perform GraphRAG sorting on representative reports. Hints include the KG path of the representative reports, classification evidence, and structural similarity suggestions, guiding the LLM to make decisions based on a three-layer KG context following specified sorting instructions. Then, non-representative reports are retrieved using a sequence expansion function in a specified script. A round-robin cross-sorting strategy is employed to cross-arrange representative and non-representative reports, deduplicating the sorted results to achieve diverse sequence coverage, and outputting the final priority review sequence.

[0011] The features of this invention are:

[0012] 1. It adopts a coarse-grained three-module collaborative design, which balances processing efficiency and technical integrity. The clustering and deduplication module solves the report redundancy problem from the source, significantly reducing the token cost of subsequent LLM processing and adapting to the large-scale report processing needs of crowdsourcing scenarios.

[0013] 2. The knowledge-enabled classification module integrates entity extraction, KG expansion, and TF-IDF classification functions to achieve accurate classification and feature weighting of reports, generate interpretable classification evidence, and solve the problems of insufficient discrimination and unclear classification basis of traditional classification methods. At the same time, the caching mechanism further reduces the cost of API calls.

[0014] 3. The intelligent sorting and expansion module integrates dynamic priority derivation and Graph RAG sorting technology, incorporating defect universality into priority calculation to avoid the rigidity of traditional priority judgment. At the same time, the cross-polling expansion strategy improves sorting diversity and defect discovery efficiency. LLM, based on structured knowledge graph decision-making, reduces sorting illusions and is suitable for general crowdsourcing platforms, with good versatility and scalability. Attached Figure Description

[0015] Figure 1 This is a flowchart of the clustering and deduplication module.

[0016] Figure 2 Flowchart for the knowledge empowerment classification module.

[0017] Figure 3 This is a flowchart of the intelligent sorting module. Detailed Implementation

[0018] This invention implements a crowdsourcing test report priority ranking method based on Graph RAG and multi-stage clustering. It mainly employs key technologies such as embedding clustering, entity extraction, KG augmentation, TF-IDF weighting, and Graph RAG ranking, and is achieved through the collaborative implementation of three core modules. The specific implementation steps are as follows:

[0019] 1. Clustering and deduplication module implementation

[0020] This module is the core pre-optimization step of the entire method. It uses a specified embedding model to transform the text descriptions of all the crowdsourcing reports to be processed into high-dimensional vectors. By calculating the cosine similarity between vectors and setting a similarity threshold of 0.8, reports with similarity exceeding the threshold are grouped into a defect cluster. A representative report is selected from each cluster using the principle of random selection or the most complete feature. Non-representative reports are temporarily stored and will be processed by the intelligent sorting and expansion module.

[0021] 2. Implementation of the knowledge empowerment classification module

[0022] This module is executed in three steps, all of which are only for processing representative reports: Step 1: Entity extraction and caching. Using a large language model, based on preset entity extraction prompts, four key entities—operations, symptoms, specific pages, and UI elements—are extracted from the report text. The extracted entity information is organized in a fixed format, with each report corresponding to one entity field. The entity extraction results of all representative reports are then persistently saved as a cache file to avoid duplicate API calls. Step 2: KG expansion. The extracted specific pages, UI elements, and other entities are injected into the global knowledge graph to establish association edges between reports and entities, enabling reports to obtain contextual information. Step 3: TF-IDF matching and classification. The IDF weights of each entity feature are calculated, and based on the degree of association between the operation, symptom, and other features in the report and each defect type, a normalized matching score is obtained between the report and each defect type. The defect type corresponding to the highest score is selected as the classification result, and the top three high IDF features are extracted as classification evidence.

[0023] 3. Implementation of the intelligent sorting and expansion module

[0024] This module is executed in two steps: The first step, Graph RAG sorting, organizes the KG path, classification evidence, structural similarity hints, and global distribution statistics of representative reports into sorting hints that LLM can understand, guiding LLM to sort representative reports according to specified sorting rules and generating a sorted sequence of representative reports; The second step, cross-expansion, retrieves reports in the same cluster based on the correspondence between representative and non-representative reports saved in the clustering stage, and uses a round-robin cross-expansion strategy to cross-arrange representative and non-representative reports, deduplicating the expanded sequence to ensure that each report appears only once, and finally outputs a complete priority sorting sequence.

[0025] The crowdsourcing test report priority ranking method based on Graph RAG and multi-stage clustering of this invention mainly includes:

[0026] Clustering and Deduplication Module: This module uses the Embedding model to cluster and deduplicat reports and filter representative reports. It records the total number of original reports for each cluster, temporarily stores non-representative reports, reduces subsequent processing costs, and suppresses interference from duplicate reports.

[0027] Knowledge-enabled classification module: Integrates entity extraction, caching, KG expansion and TF-IDF classification functions to generate accurate classification results and classification evidence, providing data support for ranking and improving classification interpretability;

[0028] Intelligent sorting module: Combining Graph RAG sorting and cross-polling expansion strategies, it generates a highly diverse and low-redundancy final sorted sequence, reducing sorting illusions and improving the efficiency of crowdsourced test report review and defect discovery capabilities.

Claims

1. A crowdsourced test report prioritization method driven by a knowledge-enhanced large model, characterized in that: This method is used to cluster, classify, and sort natural language test reports to generate a review sequence with high defect coverage, low redundancy, and high diversity. It is achieved through the collaborative efforts of three core modules, and the working steps are as follows: (1) Receive a large number of natural language test reports provided by the crowdsourcing platform and perform text formatting; (2) Clustering and deduplication module: The Embedding model is used to convert the text descriptions of all reports into high-dimensional vectors, calculate the cosine similarity between reports, cluster reports with similarity exceeding a preset threshold into a cluster, select a representative report from each cluster, record the total number of original reports in each cluster, and temporarily store non-representative reports. (3) Knowledge empowerment classification module execution processing: (3.1) Only the representative reports selected in step (2) are processed, and four key entities of operation, symptoms, specific page and UI element are extracted and cached using the large language model; (3.2) Inject the extracted entities into the global knowledge graph to establish the relationship between the report and the entities; (3.3) Calculate the IDF weight of each entity feature, achieve accurate classification of report and defect type through TF-IDF matching and scoring, and extract the top 3 high IDF features as classification evidence; (4) Intelligent sorting and expansion module execution: (4.1) Use LLM to perform Graph RAG ranking on representative reports, with prompts including the KG path of the representative reports, classification evidence and structural similarity prompts; (4.2) Call the sequence expansion function to retrieve non-representative reports, use a polling cross-sorting strategy to cross-sort representative reports and non-representative reports, perform deduplication on the sorting results, and output the final priority review sequence.

2. The method according to claim 1, characterized in that: The clustering deduplication module automatically merges and deduplicates test reports through vector representation and similarity calculation. It can dynamically adjust the clustering threshold according to the crowdsourcing scenario and select representative reports within the cluster according to the report completeness or random method, while temporarily storing non-representative reports for complete restoration during the sorting stage.

3. The method according to claim 1, characterized in that: The knowledge-enabled classification module extracts core defect entities from the test report using a large language model, integrates the extraction results into a global knowledge graph to enhance semantic association, and extracts high-discrimination features based on the entities and relationships constructed from the knowledge graph as the basis for subsequent ranking.

4. The method according to claim 1, characterized in that: The intelligent sorting and unfolding module sets basic weights based on the severity of defects, calculates comprehensive priorities by combining the number of reports within a cluster, realizes knowledge-enhanced sorting based on knowledge graph paths and classified evidence, and uses a polling cross-unfolding strategy to restore the complete report sequence, ensuring the diversity and coverage of the review sequence.

Images