Software configuration item test case data processing analysis method based on large language model

Abstract

The invention discloses a software configuration item test case data processing and analyzing method based on a large language model, which belongs to the technical field of software testing, and comprises the following steps: collecting a demand document and a design document related to a software configuration item, and integrating into a software project document; preprocessing a demand document and a design document in the software project document to obtain a software processing document; performing feature extraction, including text feature extraction and relation feature extraction, on the software processing document to obtain a fusion feature vector; performing natural language processing on the input fusion feature vector by utilizing a pre-selected large language model based on pre-trained parameters and a learned language mode, and generating a natural language description result about the configuration item and the test case through forward propagation of the model; analyzing the generated natural language description result into test case data according to a preset format specification; the test case data is optimized by evaluating the quality of the test case data and adjusting the priority of the test case data.

Description

Technical Field

[0001] The present invention relates to the technical field of software testing, and in particular to a method for processing and analyzing software configuration item test case data based on a large language model. Background Art

[0002] As software systems become increasingly complex, the testing process during their development and maintenance has always been a vital task to ensure the quality and reliability of software products. As an important part of software products, comprehensive and efficient testing of software configuration items is a key link to ensure the overall performance of the software. However, most of the existing methods for processing and analyzing software configuration item test case data rely on manual writing of test cases and simple rule-based analysis. They lack the application of automated testing technology and fail to optimize the quality and priority of software configuration item test cases, resulting in the inability to optimally configure test resources, affecting the overall effect and progress of software testing. Therefore, a software configuration item test case data processing and analysis method based on a large language model is needed to solve the above-mentioned problems. Summary of the invention

[0003] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a method for processing and analyzing software configuration item test case data based on a large language model to solve the problems raised in the above background technology.

[0004] The purpose of the present invention can be achieved by the following technical solution: A method for processing and analyzing software configuration item test case data based on a large language model comprises the following steps: Step 1: Collect the requirement documents and design documents related to the software configuration items and integrate them into software project documents; pre-process the requirement documents and design documents in the software project documents to obtain software processing documents; Step 2: extracting features from the preprocessed software-processed documents, including text feature extraction and relationship feature extraction, to obtain a fused feature vector; Step 3: Use the pre-selected large language model to perform natural language processing on the input fusion feature vector based on its pre-trained parameters and learned language patterns, and generate natural language description results about configuration items and test cases through forward propagation of the model; Step 4: Parse the natural language description results generated based on the large language model into test case data according to the preset format specification; optimize the test case data by evaluating the quality of the test case data and adjusting its priority.

[0005] As a further solution of the present invention, the requirement document and the design document include configuration items and test cases described in text.

[0006] As a further solution of the present invention, in step 1, the process of preprocessing the software project document to obtain the software processing document includes: Traverse the software project documents to obtain the text data of the requirements documents and design documents, use regular expression technology to remove HTML tags and special characters in the text data, and perform noise processing to ensure the purity of the text data; De-duplicate text data. For structured text data, compare all fields in the text data. If they are exactly the same, they are considered duplicate data and deleted. For unstructured text data, calculate the hash value of the text data to get the corresponding hash value. If there is text data with the same hash value, it is considered duplicate data and deleted.

[0007] As a further solution of the present invention, in step 2, the process of extracting features from the software-processed document includes: Split the software processing document into several words or phrases, and convert them into vector representations using word embedding technology to obtain text features; Based on the relationships between software configuration items in the software processing document, including dependency relationships and interaction relationships, a software configuration item relationship graph is constructed; each configuration item is represented as a node in the graph, and a unique identifier is used to name the node; according to the relationships between the configuration items, edges are added between corresponding nodes; wherein the edges have directions and weights; The PageRank algorithm variant is used to extract the relationship and importance of configuration items in the software configuration item relationship graph and obtain the relationship features; The obtained text features and relational features are encoded and concatenated to obtain a fused feature vector.

[0008] As a further solution of the present invention, the implementation steps of the PageRank algorithm variant include: Assign an initial weight value to each node; the initial weight value can be set to 1 or other equal values; Based on the direction of the node, obtain the node's in-edge and out-edge conditions, and update the node's weight according to the preset rules; Among them, the weight update formula of node v is expressed as:

[0009] Where PR(v) is the weight of node v; PR(u) is the weight of node u; d is the damping coefficient, which is usually 0.85; In(v) is the set of nodes pointing to node v; Out(u) is the number of outgoing edges of node u; u is any node in the set of nodes pointing to node v; Repeat the above calculation process until the preset number of iterations is reached; Based on the final calculation, all node weights are obtained to determine the importance ranking of the configuration items.

[0010] As a further solution of the present invention, the forward propagation includes: The fused feature vector is used as the input sequence and enters the embedding layer of the model; the embedding layer is used to convert the input sequence into a vector representation in a high-dimensional space; The input sequence processed by the embedding layer is input into multiple hidden layers for further abstraction and feature extraction of the input sequence. The attention mechanism is introduced to measure the correlation between the elements in each input sequence and the current hidden layer state by calculating the attention score, and the elements in the input sequence are weighted and summed according to the calculated attention score to obtain a weighted context vector as the input of the next hidden layer. The decoder uses the state of the hidden layer and the weighted context vector provided by the attention mechanism to predict the probability distribution of the next word or phrase. Finally, the predicted probability distribution is normalized by the Softmax function to obtain the final output result, which is the natural language description of the configuration items and test cases.

[0011] As a further solution of the present invention, in step 4, the process of evaluating the quality of test case data includes: By calling the software's API interface, the test case is executed to check whether the test result meets expectations; by inputting the input data of the test case, it is verified whether the actual output result is consistent with the expected output. If the actual output result is consistent with the expected output, it means that the test result meets expectations; Count the number of test cases whose test results meet expectations, and calculate the pass rate of the test cases to evaluate the test case data quality; the pass rate is determined by calculating the ratio of the number of test cases whose test results meet expectations to the total number of test cases; A normal threshold is set and the pass rate of the test case is compared with the normal threshold. If the pass rate is ≥ the normal threshold, it means that the quality of the current test case data meets the software testing standards; otherwise, the large language model is used to generate new test case data that meets the software testing standards.

[0012] As a further solution of the present invention, the process of adjusting the priority of the test case data includes: Divide the test cycle into multiple fixed time periods. In each time period, record the number of executions of each test case and calculate the execution frequency of each test case in each time period. The calculation formula is:

[0013] In the formula, PL is the execution frequency; ZC is the number of executions; ZT is the total duration of the time period; By calculating the execution frequency of each test case in each time period and taking the average value, we can get the average execution frequency of each test case in the test cycle. Prioritize each test case based on its average execution frequency from high to low.

[0014] Compared with the existing solutions, the present invention achieves the following beneficial effects: The present invention collects the requirement documents and design documents related to the software configuration items and integrates them into software project documents; pre-processes the software project documents to quickly obtain software processing documents, which greatly improves the efficiency of the preliminary preparation work of data processing and provides a data basis for the subsequent large language model; The present invention extracts features from software-processed documents, including text feature extraction and relationship feature extraction, to obtain a fused feature vector, thereby more comprehensively capturing important information in the document. Compared with the traditional manual extraction method, the method is faster and more accurate. This aspect uses a pre-selected large language model to generate natural language description results about configuration items and test cases. The large language model gives a more accurate description based on the context and existing language knowledge, avoiding the possible understanding bias of traditional methods. The present invention optimizes test case data by evaluating the test case data quality and adjusting its priority, and can discover problems in the data, such as data missing, inconsistency, inaccuracy, etc. This comprehensive evaluation helps to timely discover and solve data quality problems, help to more efficiently perform software testing, and reasonably allocate test resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The present invention will be further described below in conjunction with the accompanying drawings.

[0016] Figure 1 This is a flowchart of the software configuration item test case data processing and analysis method based on a large language model proposed in the present invention. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0018] like Figure 1 As shown, the present invention is a method for processing and analyzing software configuration item test case data based on a large language model, comprising the following steps: Step 1: Collect the requirement documents and design documents related to the software configuration items and integrate them into software project documents; pre-process the requirement documents and design documents in the software project documents to obtain software processing documents; Among them, the requirement document and design document contain configuration items and test cases described in text; In step 1, the software project document is preprocessed to obtain the software processing document, including: Traverse the software project documents to obtain the text data of the requirements documents and design documents, use regular expression technology to remove HTML tags and special characters in the text data, and perform noise processing to ensure the purity of the text data; De-duplicate text data. For structured text data, compare all fields in the text data. If they are exactly the same, they are considered duplicate data and deleted. For unstructured text data, calculate the hash value of the text data to get the corresponding hash value. If there is text data with the same hash value, it is considered duplicate data and deleted. Step 2: extracting features from the preprocessed software-processed documents, including text feature extraction and relationship feature extraction, to obtain a fused feature vector; In step 2, the process of extracting features from the software-processed document includes: Split the software processing document into several words or phrases, and convert them into vector representations using word embedding technology to obtain text features; Taking Word2 Vec as an example, the vector representation of each word after splitting is obtained through the existing training model, and the weighted average of all word vectors in the text is performed to obtain the vector representation of the text; among them, Word2 Vec is a language model for obtaining word vectors; Based on the relationships between software configuration items in the software processing document, including dependency relationships and interaction relationships, a software configuration item relationship graph is constructed; each configuration item is represented as a node in the graph, and a unique identifier is used to name the node; according to the relationships between configuration items, edges are added between corresponding nodes; wherein the edges have directions and weights, which respectively represent the direction of dependency or interaction and the importance of the relationship; It should be noted that a dependency means that the function or normal operation of one configuration item depends on the value or existence of another configuration item; for example, the database connection configuration depends on the database driver configuration; An interactive relationship refers to a relationship in which data is exchanged or mutually influenced between two configuration items; for example, there is an interactive relationship between the front-end request configuration and the back-end interface configuration; The PageRank algorithm variant is used to extract the relationship and importance between configuration items in the software configuration item relationship graph to obtain the relationship features; wherein the PageRank algorithm variant is a link analysis algorithm used to evaluate the importance of nodes; The implementation steps of the PageRank algorithm variant include: Assign an initial weight value to each node; the initial weight value can be set to 1 or other equal values; Based on the direction of the node, obtain the node's in-edge and out-edge conditions, and update the node's weight according to the preset rules; Among them, the weight update formula of node v is expressed as:

[0019] Where PR(v) is the weight of node v; PR(u) is the weight of node u; d is the damping coefficient, which is usually 0.85; In(v) is the set of nodes pointing to node v; Out(u) is the number of outgoing edges of node u; u is any node in the set of nodes pointing to node v; Repeat the above calculation process until a preset number of iterations is reached; wherein the number of iterations is set by experts in the field according to the size and structural complexity of the software configuration item relationship diagram; Based on the final calculation, all node weights are obtained to determine the importance ranking of configuration items; Encode and concatenate the obtained text features and relational features to obtain a fused feature vector; Step 3: Use the pre-selected large language model to perform natural language processing on the input fusion feature vector based on its pre-trained parameters and learned language patterns, and generate natural language description results about configuration items and test cases through forward propagation of the model; In step three, the pre-selected large language model is selected according to the actual software project requirements, such as GPT-4, BERT, LLaMA, and is deployed in the cloud to ensure that the model has the required natural language processing and generation capabilities; wherein the pre-selected large language model is an existing model, and the specific training process is not described in detail here; It should be noted that GPT-4 (Generative Pre-trained Transformer 4) is a decoder-only large language model developed by Open AI. It is based on the Transformer architecture and adopts a unidirectional generation method, that is, predicting the next word from left to right; BERT (Bidirectional Encoder Representations from Transformers) is a bidirectional encoder model based on Transformer proposed by Google. It understands semantics by considering the left and right contexts in a sentence at the same time, so as to better capture the meaning of words in different contexts; LLaMA (Large Language Model Meta AI) is a large language model developed by Meta AI. It is based on the Transformer architecture and uses a variety of optimization techniques to improve the performance and training efficiency of the model; Wherein, the forward propagation includes: The fused feature vector is used as the input sequence and enters the embedding layer of the model; the embedding layer is used to convert the input sequence into a vector representation in a high-dimensional space; Among them, the embedding layer contains linear transformation and nonlinear activation function, which can capture the semantic relationship and contextual information between word vectors; The input sequence processed by the embedding layer is input into multiple hidden layers for further abstraction and feature extraction of the input sequence. The attention mechanism is introduced to measure the correlation between the elements in each input sequence and the current hidden layer state by calculating the attention score, and the elements in the input sequence are weighted and summed according to the calculated attention score to obtain a weighted context vector as the input of the next hidden layer. The decoder uses the state of the hidden layer and the weighted context vector provided by the attention mechanism to predict the probability distribution of the next word or phrase. Finally, the predicted probability distribution is normalized by the Softmax function to obtain the final output result, which is the natural language description of the configuration items and test cases. For example, for the task of generating a test case description, the model will predict the next most likely word or phrase based on the previous configuration item information and the partial description content that has been generated, until a complete natural language description result of the configuration item and test case is generated.

[0020] Step 4: Parse the natural language description results generated based on the large language model into test case data according to the preset format specification; optimize the test case data by evaluating the quality of the test case data and adjusting its priority; The test case data includes but is not limited to title, input data, expected output, and test steps; In step 4, the process of evaluating the quality of test case data includes: By calling the software's API interface, the test case is executed to check whether the test result meets expectations; by inputting the input data of the test case, it is verified whether the actual output result is consistent with the expected output. If the actual output result is consistent with the expected output, it means that the test result meets expectations; Count the number of test cases whose test results meet expectations, and calculate the pass rate of the test cases to evaluate the test case data quality; the pass rate is determined by calculating the ratio of the number of test cases whose test results meet expectations to the total number of test cases; Set a normal threshold and compare the pass rate of the test case with the normal threshold. If the pass rate is ≥ the normal threshold, it means that the quality of the current test case data meets the software testing standard. Otherwise, continue to use the large language model to generate new test case data that meets the software testing standard. The normal threshold is set by experts in the field according to the software testing standards in actual applications. The process of prioritizing test case data includes: Divide the test cycle into multiple fixed time periods. In each time period, record the number of executions of each test case and calculate the execution frequency of each test case in each time period. The calculation formula is:

[0021] In the formula, PL is the execution frequency; ZC is the number of executions; ZT is the total duration of the time period; By calculating the execution frequency of each test case in each time period and taking the average value, we can get the average execution frequency of each test case in the test cycle. Prioritize each test case based on its average execution frequency from high to low.

[0022] In the embodiment of the present invention, by introducing the large language model technology, the automatic processing and analysis of software configuration item test case data is realized, which effectively improves the efficiency and quality of software testing, reduces the errors and costs caused by manual intervention, and provides strong support for quality assurance in the software development process.

[0023] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0024] In the several embodiments provided by the present invention, it should be understood that the disclosed system can be implemented in other ways. For example, the above-described embodiments of the invention are only illustrative, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation.

[0025] The modules described as separate accessories may or may not be physically separated, and the accessories shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0026] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0027] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0028] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for processing and analyzing software configuration item test case data based on a large language model, characterized in that: The following steps are involved: Step 1: Collect the requirement documents and design documents related to the software configuration items and integrate them into software project documents; pre-process the requirement documents and design documents in the software project documents to obtain software processing documents; Step 2: extracting features from the preprocessed software-processed documents, including text feature extraction and relationship feature extraction, to obtain a fused feature vector; Step 3: Use the pre-selected large language model to perform natural language processing on the input fusion feature vector based on its pre-trained parameters and learned language patterns, and generate natural language description results about configuration items and test cases through forward propagation of the model; Step 4: Parse the natural language description results generated based on the large language model into test case data according to a preset format specification; Optimize test case data by evaluating its quality and adjusting its priority.

2. The method for processing and analyzing software configuration item test case data based on a large language model according to claim 1, characterized in that: Requirements documents and design documents contain textual descriptions of configuration items and test cases.

3. The method for processing and analyzing software configuration item test case data based on a large language model according to claim 2, characterized in that: In step 1, the software project document is preprocessed to obtain the software processing document, including: Traverse the software project documents to obtain the text data of the requirements documents and design documents, use regular expression technology to remove HTML tags and special characters in the text data, and perform noise processing to ensure the purity of the text data; De-duplicate text data. For structured text data, compare all fields in the text data. If they are exactly the same, they are considered duplicate data and deleted. For unstructured text data, calculate the hash value of the text data to get the corresponding hash value. If there is text data with the same hash value, it is considered duplicate data and deleted.

4. The method for processing and analyzing software configuration item test case data based on a large language model according to claim 3, characterized in that: In step 2, the process of extracting features from the software-processed document includes: Split the software processing document into several words or phrases, and convert them into vector representations using word embedding technology to obtain text features; Based on the relationships between software configuration items in the software processing document, including dependency relationships and interaction relationships, a software configuration item relationship graph is constructed; each configuration item is represented as a node in the graph, and a unique identifier is used to name the node; according to the relationships between the configuration items, edges are added between corresponding nodes; wherein the edges have directions and weights; The PageRank algorithm variant is used to extract the relationship and importance of configuration items in the software configuration item relationship graph and obtain the relationship features; The obtained text features and relational features are encoded and concatenated to obtain a fused feature vector.

5. The method for processing and analyzing software configuration item test case data based on a large language model according to claim 4, characterized in that: The implementation steps of the PageRank algorithm variant include: Assign an initial weight value to each node; the initial weight value can be set to 1 or other equal values; Based on the direction of the node, obtain the node's in-edge and out-edge conditions, and update the node's weight according to the preset rules; Among them, the weight update formula of node v is expressed as: Where PR(v) is the weight of node v; PR(u) is the weight of node u; d is the damping coefficient, which is usually 0.85; In(v) is the set of nodes pointing to node v; Out(u) is the number of outgoing edges of node u; u is any node in the set of nodes pointing to node v; Repeat the above calculation process until the preset number of iterations is reached; Based on the final calculation, all node weights are obtained to determine the importance ranking of the configuration items.

6. The method for processing and analyzing software configuration item test case data based on a large language model according to claim 1, characterized in that: The forward propagation includes: The fused feature vector is used as the input sequence and enters the embedding layer of the model; the embedding layer is used to convert the input sequence into a vector representation in a high-dimensional space; The input sequence processed by the embedding layer is input into multiple hidden layers for further abstraction and feature extraction of the input sequence. The attention mechanism is introduced to measure the correlation between the elements in each input sequence and the current hidden layer state by calculating the attention score, and the elements in the input sequence are weighted and summed according to the calculated attention score to obtain a weighted context vector as the input of the next hidden layer. The decoder uses the state of the hidden layer and the weighted context vector provided by the attention mechanism to predict the probability distribution of the next word or phrase. Finally, the predicted probability distribution is normalized by the Softmax function to obtain the final output result, which is the natural language description of the configuration items and test cases.

7. The method for processing and analyzing software configuration item test case data based on a large language model according to claim 6, characterized in that: In step 4, the process of evaluating the quality of test case data includes: By calling the software's API interface, the test case is executed to check whether the test result meets expectations; by inputting the input data of the test case, it is verified whether the actual output result is consistent with the expected output. If the actual output result is consistent with the expected output, it means that the test result meets expectations; Count the number of test cases whose test results meet expectations, and calculate the pass rate of the test cases to evaluate the quality of the test case data; the pass rate is determined by calculating the ratio of the number of test cases whose test results meet expectations to the total number of test cases; Set a normal threshold and compare the pass rate of the test case with the normal threshold. If the pass rate ≥ the normal threshold, it means that the quality of the current test case data meets the software testing standards; otherwise, continue to use the large language model to generate new test case data that meets the software testing standards.

8. The method for processing and analyzing software configuration item test case data based on a large language model according to claim 7, characterized in that: The process of prioritizing test case data includes: Divide the test cycle into multiple fixed time periods. In each time period, record the number of executions of each test case and calculate the execution frequency of each test case in each time period. The calculation formula is: In the formula, PL is the execution frequency; ZC is the number of executions; ZT is the total duration of the time period; By calculating the execution frequency of each test case in each time period and taking the average value, we can get the average execution frequency of each test case in the test cycle. Prioritize each test case based on its average execution frequency from high to low.

Images