A method for building a large model in the field of software testing

Abstract

The present invention relates to a method for constructing a large model in the field of software testing. The method includes determining a base large model, constructing a software testing domain knowledge base, and pre-training the base large model based on the software testing domain knowledge base; obtaining a test instruction, querying the multimodal data related to the test instruction in the software testing domain knowledge base, inputting the test instruction and the multimodal data into the pre-trained base large model to obtain software test data; and retraining the large model based on a discriminator and a mutual information network according to the software test data. The present invention can realize the large-scale automatic generation of software test data and test codes, and improve the efficiency, adequacy and intelligence level of software testing.

Description

Technical Field

[0001] The present invention relates to the technical field of model construction, and more particularly to a method for constructing a large model in the field of software testing. Background Art

[0002] As software scale and complexity grow, software defects also increase, and the harm caused by defects becomes more and more serious. Hidden defects in software will derive more problems as the software is iterated and upgraded, resulting in more serious consequences after the defects break out.

[0003] Software testing is used to verify software functional performance, discover software problems, and check whether the software design and implementation meet the requirements and whether it meets the operation or business usage requirements. It is an important means to discover software defects and verify whether it meets the requirements. It is an important way to ensure and improve software quality. Its efficiency and effectiveness are directly related to the cost of software development and the stability of the product.

[0004] However, traditional software testing methods often rely on manually designed test cases, which is not only time-consuming, labor-intensive and costly, but also difficult to fully cover all possible test scenarios, seriously affecting the effective exposure of software defects and hindering software quality improvement and enhancement.

[0005] At present, big models have gradually been applied to software testing due to their powerful computing and learning capabilities. They are used in aspects such as automatic generation of test cases and defect mining prediction. However, in the field of software testing, the depth of knowledge is still obviously insufficient, and there are still problems such as low intelligence and low test coverage. Therefore, it is necessary to evolve from general big models to vertical field big models.

[0006] Therefore, it is particularly important to develop an efficient and automated software testing model. Summary of the invention

[0007] In view of this, the present invention provides a method for constructing a large model in the field of software testing, aiming to construct a large model that can improve the accuracy and efficiency of test data generation, and enhance the efficiency and intelligence level of software testing.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] A method for constructing a large model in the field of software testing, characterized in that:

[0010] Determine the base model, build a software testing domain knowledge base, and pre-train the base model based on the software testing domain knowledge base;

[0011] Obtain test instructions, query the multimodal data related to the test instructions in the software testing domain knowledge base,

[0012] Input the test instructions and multimodal data into the pre-trained base model to obtain software test data;

[0013] According to the software test data, the large model is retrained based on the discriminator and the mutual information network.

[0014] Preferably, the method for constructing a software testing domain knowledge base includes:

[0015] Collect original samples in the field of software testing, including command text data, visible light image data, infrared image data, and video data;

[0016] Split the original sample and convert it into a vector.

[0017] Preferably, the multimodal data is preprocessed before being input into the large model, including:

[0018] After cropping and scaling the image, generate visual tokens through encoding;

[0019] The text is segmented and cleaned, and converted into a single embedding vector to obtain text tokens;

[0020] And merge visual tokens and text tokens.

[0021] Preferably, the test instruction includes two parts, one part is determined by the type of the original data stream, and the other part is a parameter related to the generated test data.

[0022] Preferably, the objective function of the discriminator is:

[0023] D=maxlogD real +log(1-D fake )

[0024] Where D real represents the output result of the discriminator based on the real test data corresponding to the test instruction, D fake Represents the output of the discriminator based on the test data generated by the large model.

[0025] Preferably, the objective function of the mutual information network is:

[0026] Q = maxc·logQ(c|D fake )

[0027] Wherein, c represents the test instruction.

[0028] It can be seen from the above technical solution that the present invention discloses a method for constructing a big model in the field of software testing. Compared with the prior art, the present invention adopts the "base model + domain knowledge base" method, takes a general big model as the base model, and uses software testing domain knowledge data to construct a domain knowledge base, thereby building and generating a big model in the field of software testing.

[0029] The present invention can solve the problems existing in current software testing, such as weak testing error detection capability and heavy reliance on manually designed test data. Furthermore, for multimodal operation scenarios, it can realize large-scale automatic generation of software test data and test codes, thereby improving software testing efficiency, adequacy and intelligence, as well as improving the quality and reliability of software systems, and providing strong technical capability guarantee for carrying out software testing work. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0031] Figure 1 A flow chart of the method for constructing a large model in the field of software testing provided by the present invention;

[0032] Figure 2 The present invention provides a flowchart for constructing a big model in the software testing field based on the Huawei Pangu big model. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] The present invention constructs a large model in the software testing domain through "base model + domain knowledge base", in which a fused multimodal large model is used as the base model.

[0035] According to the specific test task, the context of the model input is enhanced by retrieving the test domain knowledge base, so that the large model can complete the large-scale automatic generation of software test data and test code for multimodal task scenarios. The solution disclosed in this application only needs to be built on the base model. Compared with the existing technology, the required data and computing power are far less than building a large model from scratch.

[0036] Specifically, the method for constructing a large model in the field of software testing disclosed in the embodiment is as follows: Figure 1 , including the following steps:

[0037] Determine the base model, build a software testing domain knowledge base, and pre-train the base model based on the software testing domain knowledge base;

[0038] Obtain test instructions, query the multimodal data related to the test instructions in the software testing domain knowledge base,

[0039] Input the test instructions and multimodal data into the pre-trained base model to obtain software test data;

[0040] According to the software test data, the large model is retrained based on the discriminator and the mutual information network.

[0041] In one embodiment,

[0042] The main purpose of building a software testing domain knowledge base is to provide more domain knowledge for the base model. When an instruction is input, if there is no context information, the base model may not be able to give a targeted output. By building a knowledge base, the instructions can be retrieved in the knowledge base, and the relevant knowledge points can be input to the base model, which is equivalent to telling the base model the idea of ​​solving the problem, so that the base model's answer is more targeted.

[0043] The construction of the domain knowledge base includes three steps: software testing domain knowledge collection, knowledge splitting and knowledge vectorization;

[0044] Software testing domain knowledge collection: In this embodiment, the software testing domain knowledge is the original sample of multimodal data, including system command text data, visible light image data, infrared image data, video data, etc.;

[0045] Knowledge splitting: For text data, the text to be split is split into strings of a certain length according to special characters. The length should support the maximum length of the vector model, while also considering the integrity of the semantics;

[0046] Knowledge vectorization: This embodiment uses a vectorization model to convert the split data into vectors for easy retrieval. When there are relevant test instructions to the large model, the test instructions can be vectorized and directly searched in the knowledge base, thereby improving retrieval efficiency.

[0047] Pre-training can be done by supervised fine-tuning and self-supervised training, or by superimposed training. If superimposed training is required, a common approach is to first perform self-supervised training and then continue with supervised fine-tuning. This can further improve the performance of the model based on the existing task knowledge and adapt it to more specific tasks.

[0048] In one embodiment, a test instruction is obtained, and multimodal data related to the test instruction is queried in a software testing domain knowledge base;

[0049] In this embodiment, after constructing the domain knowledge base, it is necessary to construct a knowledge search algorithm to quickly retrieve content related to the input. Preferably, the present application adopts the HNSW algorithm, that is, using a multi-level graph to store data points, and by constructing highly connected subgraphs, points with similar features form tight clusters.

[0050] During the search process, starting from the high-level node, by comparing the distance between the query point and the current node, the node with the closest distance is selected and the search is gradually recursively performed toward the lower levels until the nearest neighbor is found or the bottom layer is reached.

[0051] In the present invention, the HNSW algorithm also uses optimization techniques such as branch reduction, jump and sorting to reduce the search path and improve the search efficiency.

[0052] In the software testing scenario, each node can be regarded as the data vector mentioned above. By using this method, the knowledge data most relevant to the input can be queried. By efficiently retrieving the knowledge base, more accurate and targeted support can be provided for the intelligent output of large models.

[0053] In a preferred embodiment, multimodal data needs to be screened and preprocessed before being input into the large model.

[0054] Generally, the quality of data generated by a large multimodal model is limited by the quality and quantity of the training data. If the amount of training data is insufficient or noisy, the quality of the generated data may be affected. Moreover, if the training data does not cover various scenarios and situations well, the generated test data may be biased and fail to fully reflect the diversity of the operating environment of the object under test. Due to the particularity of terrain and scenes, general large models lack such data training. Therefore, it is necessary to screen samples specifically based on the particularity of terrain and scenes, so as to optimize the model in a targeted manner.

[0055] Further, in this embodiment, the preprocessing includes:

[0056] First, the image is cropped and scaled, and the text is segmented and cleaned; these processed data can be regarded as a series of tokens to represent the original data;

[0057] Secondly, for each type of modality processed data, use a specific encoder to encode it and generate corresponding tokens. That is, for image data, use a convolutional neural network to encode it and generate visual tokens. For text data, use text embedding technology to convert text into word embedding vectors and generate text tokens. Other types of data can also be encoded using corresponding encoders to generate corresponding tokens;

[0058] Then, the tokens of different modalities are fused as the input of the large model so that the model can process data of multiple modalities at the same time. Token fusion can be achieved in many ways, such as concatenation, splicing, weighted averaging, etc., among which concatenation and splicing are preferred.

[0059] In one embodiment, the test instructions and multimodal data are input into a pre-trained base large model to obtain software test data; according to the software test data, the large model is retrained based on the discriminator and the mutual information network.

[0060] After generating test data and test code based on the big model, this application evaluates the test data and test code through the InfoGAN-based discrimination module, thereby fine-tuning and retraining the domain big model to generate test data for the environment.

[0061] The input of the large model in the present invention consists of multimodal data and test instructions c;

[0062] Furthermore, the output data of the large model is used as the input of the discriminator. At the same time, the discriminator receives the real test data corresponding to the test instructions.

[0063] That is, in the discrimination part, the discriminator will output D according to the real test data real , and select relevant multimodal data and test instructions according to the distribution, and obtain the test data output D through the large model fake ; So the objective function of constructing the discriminator is:

[0064] D=maxlogD real +log(1-D fake )

[0065] Where D real represents the output result of the discriminator based on the real test data corresponding to the test instruction, D fake Represents the output of the discriminator based on the test data generated by the large model. By maximizing the output of the discriminator, the real data and the generated test data can be accurately distinguished.

[0066] The objective of the base model can be further described as G = minlog(1-D fake). Minimizing the output of the large model, that is, generating test data can be closer to the real data.

[0067] In addition, the test data and test instructions generated by the large model are input into the mutual information network to output the mutual information between the test data and the control variables, thereby measuring the correlation between the prompt words input by the large model and the multimodal data finally output.

[0068] In the present invention, the mutual information network Q is used to maximize the mutual information between the control variables and the generated test data, and its objective function is Q=maxc·logQ(c|x), where x is the generated test data and c represents the test instruction; so that the generated test data can better meet the requirements.

[0069] In one embodiment, the loss function of the algorithm can be described as the sum of three parts, namely, the loss function of the discriminator, the loss function of the generative model, and the loss function of the mutual information network.

[0070]

[0071] After the networks are trained to converge quickly, a large model can be used to quickly generate multimodal test data, or a discriminant network can be used to discriminate the actual data input to determine whether it belongs to the actual data stream that the system should actually receive.

[0072] Furthermore, the execution process of this application is explained through examples.

[0073] This example uses Huawei Pangu Big Model as the base model to build a software testing domain knowledge base. It enhances the context of the base model input by retrieving the testing domain knowledge base, and completes the entire life cycle process from data set, model training / fine-tuning, to model deployment through the Huawei Pangu Big Model UI interface.

[0074] Specific reference Figure 2 , the steps include:

[0075] (1) Data preparation and management

[0076] When developing large models in the field of software testing, it is necessary to upload training data independently, and perform self-supervised training and supervised fine-tuning based on the platform's preset models and algorithms. The quality of the data will affect the accuracy of the model, and a large amount of high-quality data is more likely to train a high-precision model.

[0077] The datasets used for supervised fine-tuning are divided into two categories: training files and evaluation files. Both need to be saved in ".json" format. The training file is used for training models, and the evaluation file is used to verify the effect of the trained domain model when testing code output. The dataset used for self-supervised training is a ".json" format file. The name of the training data file can be customized, and the content is composed of text unlabeled data, and each line is a piece of data.

[0078] (2) Construction of Domain Knowledge Base

[0079] After completing data preparation and data set construction, the construction steps of the domain knowledge base also require knowledge splitting and knowledge vectorization. For the original samples of multimodal data in the software testing field, knowledge splitting is performed according to the data type. For text data, the text to be split is split into strings of a certain length according to special characters. This length should support the maximum length of the vector model, and the semantic integrity should also be considered. Then, the vectorized model is used to convert the split data into vectors, which is convenient for large domain model retrieval and supports the completion of model training.

[0080] (3) In the field of software testing, large model training is a process of continuous optimization and iteration. It requires continuous adjustment and optimization of the model version by verifying the test data, hyperparameters, and the model itself in order to train the best model. This process may require multiple iterations, but the ultimate goal is to improve the performance and accuracy of the model through continuous optimization.

[0081] Both supervised fine-tuning and self-supervised training can be performed independently or in combination on the basis of the pre-training phase. If superimposed training is required, a common approach is to perform self-supervised training first and then continue with supervised fine-tuning. This can further improve the performance of the model based on the existing task knowledge and adapt it to more specific tasks.

[0082] To create a new model training, you need to log in to the Pangu Big Model Open Platform. Select "Custom Workflow > Model Training" in the left navigation bar, and click "New Model" in the upper right corner of the interface;

[0083] Select the training type, pre-trained model, data set, and training parameters to create a model training task. The training parameters can control the process of model training optimization. Different parameter values ​​may affect the model training and convergence speed. When training a model, most parameters can use the default values, and the batch size and learning rate can be changed.

[0084] In supervised fine-tuning, you should ensure that the model can be saved multiple times during the training process, and that logs can be printed and the performance of the model can be evaluated in a timely manner to monitor the progress and effect of the training. It is recommended that the values ​​of the four parameters, namely, the number of training steps, the number of steps for saving the model, the log interval, and the model evaluation interval, should be kept consistent or in multiples.

[0085] (4) Domain Large Model Evaluation

[0086] After the model training is successful, the Pangu Big Model Open Platform can automatically evaluate the big model in the software testing field and provide relevant indicator data, including the BLEU indicator and the ROUGE indicator.

[0087] The BLEU indicator is mainly used to evaluate machine translation tasks. Its overall idea is accuracy. If a standard translation reference is given, the sentence generated by the large model is the candidate, the sentence length is n, and there are m words in the candidate that appear in the reference. m / n is the calculation formula of 1-gram of BLEU. BLEU has many variants and can be divided into multiple evaluation indicators based on n-gram. The common indicators are BLEU-1, BLEU-2, BLEU-3, and BLEU-4.

[0088] The ROUGE metric is a method for evaluating automatic text summarization, using a recall-based summary evaluation method. It is used to compare automatically generated test cases and codes with manually written reference cases and codes, and calculate the n-gram overlap between the two, that is, how many phrases of length n appear in both summaries.

[0089] (5) Domain-wide large model optimization and deployment

[0090] The quality of test data generated by domain-wide models is limited by the quality and quantity of training data. If the amount of training data is insufficient or noisy, the quality of generated data may be affected. If the training data does not cover a variety of scenarios and situations well, the generated test data may be biased.

[0091] After completing model training, large domain models often need to further screen samples and perform targeted tuning on the model, which usually includes operations such as model structure optimization, model fusion, and hyperparameter tuning.

[0092] When the software testing domain large model training and tuning are completed, that is, when the model is in the "Completed" state, you can start the model deployment operation. Log in to the Pangu Big Model Open Platform, select "Custom Workflow > Model Deployment" in the left navigation bar, and click "Deploy" in the upper right corner of the interface.

[0093] After the model is successfully deployed, the status changes to "Running". Users can call the model through the API and playground interface to automatically generate test cases and test codes.

[0094] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0095] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for constructing a large model in the field of software testing, characterized in that: Determine the base model, build a software testing domain knowledge base, and pre-train the base model based on the software testing domain knowledge base; Acquire the test instruction, and query the multimodal data related to the test instruction in the software testing domain knowledge base. The method for constructing the software testing domain knowledge base includes: Collect original samples in the field of software testing, including command text data, visible light image data, infrared image data, and video data; Split the original sample and convert it into a vector; Input the test instructions and multimodal data into the pre-trained base model to obtain software test data; According to the software test data, the large model is retrained based on the discriminator and the mutual information network.

2. A method for constructing a large model in the field of software testing according to claim 1, characterized in that: Preprocess the multimodal data before inputting into the large model, including: After cropping and scaling the image, generate visual tokens through encoding; The text is segmented and cleaned, and converted into a single embedding vector to obtain text tokens; And merge visual tokens and text tokens.

3. A method for constructing a large model in the field of software testing according to claim 1, characterized in that: The test instruction consists of two parts, one part is determined by the type of the original data stream, and the other part is the parameters related to the generated test data.

4. A method for constructing a large model in the field of software testing according to claim 1, characterized in that: The objective function of the discriminator is: D=maxlogD real +log(1-D fake ) Where D real represents the output result of the discriminator based on the real test data corresponding to the test instruction, D fake Represents the output of the discriminator based on the test data generated by the large model.

5. A method for constructing a large model in the field of software testing according to claim 1, characterized in that: The objective function of the mutual information network is: Q=maxc·logQ(c|D fake ) Wherein, c represents the test instruction.

Images