Test script generation method and device and electronic equipment

Abstract

The invention provides a test script generation method and device and electronic equipment, and the method comprises the steps: determining sample data participating in model training according to multi-modal data corresponding to a plurality of applications, the multi-modal data comprises image data representing the overall layout of the application, video data representing the interaction condition of the user and voice data representing the test intention of the application; training a pre-constructed neural network model according to the sample data, adjusting parameters and a structure of the neural network model, and obtaining a target model for outputting a test instruction set of the application; processing the test instruction generation request of the target application based on the target model, and obtaining a target test instruction set corresponding to the target application; and generating a test script corresponding to the target application according to the target test instruction set. According to the method and the device, the automatic test script can be generated by utilizing the multi-modal large model architecture, the test script generation efficiency is improved, and the accuracy and the reliability of the test script are improved through the participation of the multi-modal data.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a test script generation method, device and electronic device. Background Art

[0002] Automated testing is an important part of the software development process. It can improve testing efficiency and reduce manual errors. However, the existing automated test script generation method mainly relies on professional test engineers to manually write according to test requirements, which limits the participation of non-professional users and makes it difficult to cover complex test scenarios. In more advanced methods, dynamic data can be used to map pre-written test steps to form a test script, but this method is not flexible. At present, the generation of automated test scripts is often based on single-modal data, resulting in inflexible generation of test scripts and insufficient accuracy and reliability of test scripts.

[0003] It can be seen that the existing technology has the following disadvantages when generating test scripts: the limitation of relying on professionals makes it unfriendly to non-professional users; there are deficiencies in dealing with complex and diverse testing requirements; the generation flexibility of test scripts is insufficient, and it is difficult to ensure the accuracy and reliability of test scripts. Summary of the invention

[0004] In view of the above problems, embodiments of the present application provide a test script generation method, device and electronic device that overcome the above problems or at least partially solve the above problems.

[0005] In a first aspect, an embodiment of the present application provides a test script generation method, comprising:

[0006] Determine sample data for model training based on multimodal data corresponding to multiple applications, wherein the multimodal data includes image data representing the overall layout of the application, video data representing user interaction, and voice data representing application test intent;

[0007] Training a pre-built neural network model according to the sample data, adjusting the parameters and structure of the neural network model, and obtaining a target model for outputting a test instruction set for an application;

[0008] Processing a test instruction generation request corresponding to a target application based on the target model to obtain a target test instruction set corresponding to the target application;

[0009] A test script corresponding to the target application is generated according to the target test instruction set.

[0010] In a second aspect, an embodiment of the present application provides a test script generating device, including:

[0011] A determination module, used to determine sample data participating in model training according to multimodal data corresponding to a plurality of applications, wherein the multimodal data includes image data representing the overall layout of the application, video data representing the user interaction situation, and voice data representing the application test intention;

[0012] A processing module, used to train a pre-built neural network model according to the sample data, adjust the parameters and structure of the neural network model, and obtain a target model for outputting a test instruction set for the application;

[0013] An acquisition module, configured to process a test instruction generation request corresponding to a target application based on the target model, and acquire a target test instruction set corresponding to the target application;

[0014] A generation module is used to generate a test script corresponding to the target application according to the target test instruction set.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the test script generation method described in the first aspect above.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the test script generation method described in the first aspect above are implemented.

[0017] The technical solution of the embodiment of the present application, after collecting multimodal data of multiple applications to determine sample data, adjusts the model parameters and structure of the constructed neural network model based on the sample data, determines the target model for outputting the application test instruction set, uses the target model to process the test instruction generation request corresponding to the target application, obtains the target test instruction set corresponding to the target application, and generates the test script corresponding to the target application based on the target test instruction set. It can use the multimodal large model architecture to generate automated test scripts, significantly reducing the time and cost of manual script writing, improving the efficiency of test script generation, allowing non-professionals to easily participate in the generation of test scripts, and lowering the threshold; and the accuracy and reliability of the test scripts are improved through the participation of multimodal data. At the same time, the processing method of generating scripts based on the model can adapt to changing test requirements, quickly adjust test strategies, and improve the flexibility of script generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram showing a test script generation method provided in an embodiment of the present application;

[0019] Figure 2 A specific implementation flow chart of the test script generation method provided in an embodiment of the present application is shown;

[0020] Figure 3 A schematic diagram showing a test script generating device provided in an embodiment of the present application;

[0021] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0023] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. The multiple in the embodiments of the present application can include two and more than two.

[0024] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0025] The present application embodiment provides a test script generation method, such as Figure 1 As shown, the method includes:

[0026] Step 101: determine sample data participating in model training based on multimodal data corresponding to multiple applications, where the multimodal data includes image data representing the overall layout of the application, video data representing user interaction, and voice data representing application test intent.

[0027] In this embodiment, multimodal data corresponding to multiple applications are first collected, sample data for model training is determined based on the collected multimodal data, and then the sample data is used for model training. The multimodal data provided by the application includes image data representing the overall layout of the application, video data representing the user's interaction in the application interface, and voice data representing the application test intent. Image data includes, for example, application screenshots, interface design drawings, etc., video data is video data corresponding to the user operating the application interface, which can represent the user's interactive behavior, and voice data is the user's verbal description, which can reflect the user's test intent and needs. Among them, multiple applications may include different types of applications, and the multimodal data provided by the application is the corresponding data in the application test scenario. In the application test scenario, the application is tested based on at least one test intent.

[0028] Step 102: Train the pre-built neural network model according to the sample data, adjust the parameters and structure of the neural network model, and obtain a target model for outputting a test instruction set for the application.

[0029] After collecting multimodal data provided by multiple applications to determine sample data, the pre-constructed neural network model is trained based on the sample data. The parameters and structure of the neural network model are adjusted through model training to determine the target model based on the model training. The determined target model is a model of a test instruction set for the output application.

[0030] Among them, when constructing a neural network model, a neural network model including a modal encoder, an input projector, a multimodal fusion layer, a multimodal converter and a decoder is constructed; in the model training process, multimodal data is used as sample data to participate in the training, so that the neural network model can better understand the features of the input data of various modalities, so that the model structure and parameters can be efficiently adjusted based on the sample data while improving the model understanding efficiency, so as to improve the model training efficiency.

[0031] Step 103: Process the test instruction generation request corresponding to the target application based on the target model to obtain a target test instruction set corresponding to the target application.

[0032] After the target model is determined through model training, in response to a test instruction generation request corresponding to the target application, the test instruction generation request is processed based on the target model to generate a target test instruction set corresponding to the target application through processing of the target model, thereby utilizing the model to efficiently and accurately provide a target test instruction set adapted to the target application.

[0033] The test instruction generation request corresponding to the target application carries the test intent of the target application. By using the target model to process the test instruction generation request, a target test instruction set that matches the test intent of the target application can be output. The target test instruction set includes multiple test instructions, and the multiple test instructions indicate that the target application should be tested in accordance with the test intent.

[0034] Step 104: Generate a test script corresponding to the target application according to the target test instruction set.

[0035] After using the target model to process the target application's test instruction generation request and obtain the target test instruction set corresponding to the target application, the script generator is used to process the target test instruction set to generate a test script associated with the target application. The role of the script generator is to use the rule engine to deepen the analysis of the test instruction set to ensure that the preliminary instructions provided by the target model can be accurately converted into detailed test steps. By using the script generator to process the target test instruction set associated with the target application provided by the target model, the test instructions can be deeply analyzed to capture the intent and context of the test instructions, generate a logically clear and coherent test script, and ensure the quality of the test script generation.

[0036] The above implementation scheme of the present application, after collecting multimodal data of multiple applications to determine sample data, adjusts the model parameters and structure of the pre-built neural network model based on the sample data, determines the target model for outputting the application test instruction set, uses the target model to process the test instruction generation request corresponding to the target application, obtains the target test instruction set corresponding to the target application, and generates the test script corresponding to the target application based on the target test instruction set. It can use the multimodal large model architecture to generate automated test scripts, significantly reducing the time and cost of manual script writing, improving the efficiency of test script generation, allowing non-professionals to easily participate in the generation of test scripts, and lowering the threshold; and the accuracy and reliability of the test script are improved through the participation of multimodal data. At the same time, the processing method of generating scripts based on the model can adapt to changing test requirements, quickly adjust test strategies, and improve the flexibility of script generation.

[0037] The following is an introduction to the process of determining the target model of the output application test instruction set based on model training. The sample data in this embodiment includes a training data set, and when the pre-built neural network model is trained according to the sample data, the parameters and structure of the neural network model are adjusted, and the target model of the test instruction set for the output application is obtained, it includes:

[0038] Performing feature fusion processing and decision fusion processing on multimodal data in the training data set to generate multiple training arrays including comprehensive feature vectors and matching comprehensive test decisions;

[0039] Adjust the parameters and structure of neural network models based on multiple training arrays;

[0040] When the model performance of the neural network model meets the conditions, determining the target model;

[0041] Among them, the comprehensive feature vector includes image features for identifying user interface UI elements, video features for capturing user interactions, and voice features for understanding test intentions. Each comprehensive feature vector corresponds to a test intention, and the comprehensive test decision matching the comprehensive feature vector is adapted to the corresponding test intention.

[0042] The sample data determined based on the multimodal data provided by multiple applications includes a training data set and a verification data set. The training data set is used to participate in the model training process, and the verification data set is used to verify the trained model. The data volume corresponding to the training data set and the verification data set is a preset ratio.

[0043] When performing model training based on a training data set, feature fusion processing and decision fusion processing are performed on the multimodal data in the training data set to generate multiple training arrays through feature fusion and decision fusion. Each training array includes a comprehensive feature vector and a comprehensive test decision matching the comprehensive feature vector. The comprehensive feature vector is determined based on multimodal feature fusion, and the comprehensive test decision is determined based on test decisions of multiple modalities.

[0044] The comprehensive feature vector includes image features for identifying user interface (UI) elements, video features for capturing user interactions, and voice features for understanding test intent. Each comprehensive feature vector corresponds to a test intent, and each comprehensive feature vector corresponds to a comprehensive test decision. The comprehensive test decision is adapted to the corresponding test intent. It can be understood that the comprehensive test decision is a comprehensive decision determined based on the corresponding test intent and after considering multi-dimensional features.

[0045] After generating multiple training arrays by processing the training data set, the parameters and structure of the neural network model are adjusted based on the multiple training arrays to train the constructed neural network model using the training arrays. When it is identified that the model performance of the neural network model meets the conditions, the current neural network model is determined to be the target model.

[0046] In the above implementation process, a training data set is determined in the sample data, the multimodal data in the training data set is processed to generate multiple training arrays, and the parameters and structure of the constructed neural network model are adjusted based on the multiple training arrays to determine the target model of the test instruction set for the output application based on the model training.

[0047] Optionally, in the above embodiment, when performing feature fusion processing and decision fusion processing on the multimodal data in the training data set to generate a plurality of training arrays including comprehensive feature vectors and matching comprehensive test decisions, it includes:

[0048] For each application, based on N test intents associated with the application, determine N pieces of multimodal data from a set of multimodal data corresponding to the application, where N is an integer greater than or equal to 1;

[0049] For each piece of multimodal data, an image feature vector, a video feature vector and a text feature vector are generated based on data encoding and standardization processing. The text feature vector is determined based on the speech data representing the test intent.

[0050] Performing feature fusion according to the image feature vector, the video feature vector and the text feature vector to generate a comprehensive feature vector, and performing decision fusion based on the test decisions corresponding to the image feature vector, the video feature vector and the text feature vector to generate a comprehensive test decision;

[0051] For each piece of multimodal data, a training array is constructed based on the corresponding comprehensive feature vector and the comprehensive test decision to generate multiple training arrays for model training.

[0052] When collecting data, each application provides a set of multimodal data, and for a single application, it corresponds to N test intentions. Then, a set of multimodal data corresponding to a single application can be divided into N parts, and each multimodal data corresponds to one test intention.

[0053] For each piece of multimodal data in the training data set, image feature vectors, video feature vectors and text feature vectors are generated based on data encoding and standardization. The image feature vector is generated based on the image data, the video feature vector is generated based on the video data, and the text feature vector is generated based on the speech data that represents the test intent.

[0054] When performing data encoding and standardization, it is necessary to use the modal encoder and input projector in the multimodal large model architecture. The modal encoder is used to encode data, converting data from one form or format to another form so that the computer can recognize it; the input projector is used to standardize the output content of the modal encoder so that the output content adapts to the unified model input layer.

[0055] Modal encoders include image encoders, video encoders, and speech encoders. For image encoders, they can preprocess image data and extract features during the encoding process. When preprocessing image data, screenshots or interface design drawings are preprocessed, including image enhancement, denoising, and resolution adjustment to improve the generalization ability of the model; when extracting features, convolutional neural networks (CNN) are used to identify UI components and layouts, extract key visual features such as buttons, text boxes, drop-down menus, etc., and use image segmentation technology to further separate UI components and extract more detailed features. Images can also be annotated manually or semi-automatically at this stage.

[0056] For video encoders, we use video frame processing to break down the video into continuous frames, extract the image information of each frame, and use optical flow analysis technology to capture subtle changes in user operations in the video and identify the user's operation sequence in the video, such as clicks, slides, and inputs. Clicking behavior in the video cannot be captured visually, but features extracted from continuous frames before and after the state change can be used to identify which UI components may have been clicked. Through timing analysis technology, we can understand the order and logical relationship of operations.

[0057] For the speech encoder, the recorded user's verbal description is converted into text using speech recognition technology, and the text is parsed using natural language processing (NLP) technology to extract the user's test intention. When processing video data and speech data, manual or semi-automatic annotation can also be used.

[0058] The input projector is used to perform standardization processing on the output content of the modal encoder, for example, including feature scaling, normalization, etc. The output content of the input projector includes an image feature vector, a video feature vector, and a text feature vector.

[0059] For each multimodal data, when performing vector splicing according to its corresponding image feature vector, video feature vector and text feature vector, the image feature vector and the video feature vector are spliced, the image feature vector and the text feature vector are spliced, and the video feature vector and the text feature vector are spliced. After splicing, the features are weighted summed according to the splicing situation to form a comprehensive feature vector. When performing weighted summation of features, different weights can be assigned to the image, video, and text feature vectors according to the importance or signal-to-noise ratio of the modality. The corresponding weights after splicing the image and video, the image and text, and the video and text can be determined according to the weights corresponding to the two features, such as determining the weight corresponding to the spliced ​​features based on the mean of the two feature weights. Since the image feature vector is used to characterize UI elements, the video feature vector is used to capture user interactions, and the text feature vector is used to identify test intent, constructing a comprehensive feature vector through feature fusion can provide a comprehensive view of the test environment.

[0060] Among them, when the test decisions corresponding to the image feature vector, the video feature vector and the text feature vector are respectively fused to generate a comprehensive test decision, it includes: identifying the test decision corresponding to the image feature vector based on the image decision model, identifying the test decision corresponding to the video feature vector based on the video decision model, and identifying the test decision corresponding to the text feature vector based on the speech decision model; the test decisions of the three modalities are fused to generate a comprehensive test decision.

[0061] In this embodiment, it is necessary to pre-train independent classifiers (decision models) for each modality. For example, CNN is used to classify image features, RNN is used to classify video features, and deep learning models are used to classify text features extracted from speech. These independent classifiers are trained separately using labeled image, video, and text data, and the performance of each independent classifier is evaluated to ensure the accuracy of classification. It should be noted that for the speech decision model, it outputs a test decision based on the test intent, and data in text form is easier to process. When performing model training and model application, data in text form after speech conversion is usually used. Understand from another perspective that the essence of the speech decision model is to process text converted from speech. The reason why it is called a speech decision model is to adapt to the speech modality.

[0062] The image feature vector, video feature vector and text feature vector are processed respectively by using the classifiers of the three modalities (decision models of the three modalities). After obtaining the test decision corresponding to the image feature vector, the test decision corresponding to the video feature vector and the test decision corresponding to the text feature vector, the test decisions of different modalities are fused to form a comprehensive test decision.

[0063] For each piece of multimodal data, after determining its corresponding comprehensive feature vector and comprehensive test decision, a training array is constructed based on the comprehensive feature vector and the comprehensive test decision, and multiple training arrays participating in model training are determined according to the training arrays corresponding to the multiple pieces of multimodal data. It should be noted that for each piece of multimodal data, the process of determining its comprehensive feature vector and comprehensive test decision and then generating a training array is performed in the multimodal fusion layer.

[0064] During the above implementation process, the corresponding image feature vector, video feature vector and text feature vector are determined for each piece of multimodal data, feature fusion is used to generate a comprehensive feature vector, decision fusion is used to generate a comprehensive test decision, and a training array is constructed based on the comprehensive feature vector and the comprehensive test decision to determine multiple training arrays for model training.

[0065] Optionally, after determining the multiple training arrays, adjusting the parameters and structure of the neural network model based on the multiple training arrays includes:

[0066] Pre-training the neural network model according to the first array set so that the neural network model learns features of data of different modalities;

[0067] The neural network model is fine-tuned multiple times according to the second array set to adjust the parameters and structure of the neural network model; the multiple training arrays are divided into a first array set and a second array set.

[0068] After the multimodal fusion layer processes and determines a plurality of training arrays, the plurality of training arrays are divided into a first array set and a second array set. The first array set is used to pre-train the neural network model, so that the neural network model can learn the features of data of different modalities and learn how to parse the data through pre-training, and train the understanding ability of the neural network model, so that the neural network model can better understand the features.

[0069] After pre-training, on the basis of pre-training, the model is fine-tuned using the second array set to adapt to specific test instruction generation requirements by adjusting the parameters and structure of the neural network model, thereby improving the accuracy and generalization ability of the model.

[0070] Among them, the training array needs to be processed in both the pre-training stage and the model fine-tuning stage. Specifically, the comprehensive feature vector generated in the feature fusion stage is used as the input sequence and input into the multimodal converter. The multimodal converter adds position encoding to the comprehensive feature vector to retain the time or space information in the sequence so that the model can understand the order or position relationship between the features; the comprehensive test decision obtained in the decision fusion stage is used as the input of the multimodal converter in the form of an embedded vector, so that the model can consider the decision results of different modalities when generating the final output content.

[0071] The neural network model in this embodiment uses the Transformer architecture, and the self-attention mechanism in the Transformer architecture can learn how to establish connections between different modalities to better understand the overall scene. For example, the correspondence between action sequences and instructions in a video, or the matching of UI elements in an image with text descriptions.

[0072] The pre-training stage is used to learn general feature representations. In the pre-training stage, after taking the comprehensive feature vector and comprehensive test decision as input, data pre-processing is performed, the model architecture is adjusted, the optimization strategy is selected, and the evaluation index is selected. The fine-tuning stage is used to adjust the model parameters and structure in detail. It also requires data pre-processing, adjusting the model architecture, selecting the optimization strategy, and selecting the evaluation index.

[0073] When performing data preprocessing, long sequences are split into shorter segments to adapt to the maximum sequence length limit of the model, and random cropping, rotation or flipping are performed to improve the generalization ability of the model. For the pre-training stage, the focus is on general features, and the segmentation, random cropping, rotation, and flipping of long sequences will be emphasized; for the fine-tuning stage, cropping and segmentation will be more precise, and information important to specific tasks can be retained.

[0074] When adjusting the model architecture, in the pre-training stage, the number of Transformer layers, the number of attention heads, and the number of hidden units are adjusted to ensure the optimal configuration; in the fine-tuning stage, the structure of the pre-trained model will be fixed, and only certain specific layers will be adjusted or task-specific heads will be added.

[0075] When optimizing strategy selection, adjust the optimizer and learning rate to ensure that the model can effectively learn from multimodal data. In the pre-training stage, use a larger learning rate and slow learning rate decay; in the fine-tuning stage, use a smaller learning rate.

[0076] When selecting evaluation indicators, select the cross entropy loss function as the loss function, and define accuracy and recall as evaluation indicators. In the pre-training stage, the cross entropy loss function is generally selected, and high accuracy or recall is not pursued; in the fine-tuning stage, indicators such as accuracy and recall will be paid attention to.

[0077] During the pre-training stage and the model fine-tuning stage, the multimodal converter processes the input content, fuses the input content, and finally outputs multimodal machine code, which is then processed by the decoder.

[0078] For the decoder, the decoder generates instructions that match the input modal information through conditional generation based on the comprehensive machine code output by the multimodal converter. The decoder includes multiple decoder heads, each of which focuses on different aspects of generating test instructions, such as one head responsible for generating user operation steps and another head responsible for generating verification conditions. At this stage, it is necessary to ensure that the decoder head takes into account the previously generated content and the input multimodal data when generating instructions, so as to ensure that the generated instructions are coherent, logically clear, and meet specific test requirements.

[0079] When generating instructions, the decoder converts multimodal information into specific test actions based on rule conditions, such as generating click actions based on the position and size of UI elements; or, using predefined instruction templates, generates customized instructions based on multimodal information.

[0080] During the instruction generation phase, the generated test instructions need to be syntax and logic verified to ensure the correctness and executability of the instructions. At the same time, the test instructions need to be performance optimized, such as reducing unnecessary waiting time and optimizing resource usage.

[0081] For the decoder, its working process can be simply understood as: initialize the test environment and preset conditions according to the input content, such as determining the test platform, loading necessary test data, etc.; based on the operation sequence recognized by the video encoder, the decoder generates corresponding test instructions and processes; based on the test requirements analyzed by the speech encoder, the decoder configures the necessary test parameters and expected descriptions; based on the UI component information analyzed by the image encoder, identify and locate UI elements, understand the interaction logic between elements, and generate visual verification instructions for comparing the changes in the UI interface before and after the test execution.

[0082] After processing the input content based on the decoder, a set of test instructions corresponding to the multimodal data is output. In the pre-training stage, the output content can be used to help the model understand how to generate output data from the input data. In the fine-tuning stage, the output content is used to provide a basis for model evaluation.

[0083] Optionally, the sample data in this embodiment includes a validation data set, and when the model performance of the neural network model meets the conditions, determining the target model includes:

[0084] Evaluate the model performance of the neural network model based on the validation dataset;

[0085] In response to the model performance of the neural network model satisfying a condition, determining the neural network model as a target model;

[0086] In response to the model performance of the neural network model not satisfying the conditions, the process of adjusting the parameters and structure of the neural network model based on the training data set and evaluating the model performance based on the validation data set is repeated until the model performance of the neural network model satisfies the conditions to determine the target model.

[0087] After fine-tuning the model, the model performance of the neural network model is evaluated based on the validation data set. For the validation data set, feature fusion and decision fusion are also required to determine multiple validation arrays, and the model performance is evaluated based on multiple validation data. If the model performance evaluation result meets the conditions, the current neural network model is determined to be the target model; if the model performance evaluation result does not meet the conditions, it is necessary to continue to adjust the parameters and structure of the neural network model based on the training data set, and continue to evaluate the model performance based on the validation data set. If the evaluation result still does not meet the conditions after adjustment, it is necessary to continue the process until the model performance of the neural network model meets the conditions and the target model is determined. When evaluating the model performance of the neural network model, accuracy and recall can be considered, and model performance can be measured based on accuracy and recall. For example, for accuracy, compare the test instruction set output by the model with the real test instruction set to verify the accuracy.

[0088] It should be noted that the following strategies can also be used for model optimization: 1. Hyperparameter adjustment, find the optimal hyperparameter combination through grid search, random search or automated hyperparameter optimization technology; 2. Regularization technology, apply regularization techniques such as Dropout and weight decay to prevent model overfitting; 3. Early stopping rule, during the training process, if the performance on the validation data set has not improved for multiple consecutive cycles, stop training in advance to avoid overfitting. Furthermore, the evaluation of generated instructions can be collected through the user feedback mechanism to guide model optimization; the model can also be updated regularly based on feedback to improve the accuracy and availability of instruction generation.

[0089] In an optional embodiment of the present application, when a test instruction generation request corresponding to a target application is processed based on a target model to obtain a target test instruction set corresponding to the target application, the process includes:

[0090] In response to a test instruction generation request carrying multimodal data corresponding to a target application, the test instruction generation request is input into a target model; and a target test instruction set is obtained according to a result of the target model processing the test instruction generation request.

[0091] After obtaining a test instruction generation request carrying multimodal data corresponding to the target application, the test instruction generation request is input into the target model, and the target model processes the carried multimodal data to generate a target test instruction set corresponding to the target application based on the multimodal data provided by the target application, thereby realizing the use of a neural network model to efficiently and accurately output instructions adapted to the target application.

[0092] After the target test instruction set corresponding to the target application is outputted using the target model, generating a test script corresponding to the target application according to the target test instruction set includes:

[0093] Identify the test action set corresponding to the target test instruction set based on the rule base, and generate a test script according to the test action set; or

[0094] Determine a script generation template according to the test intent of the target application and the instruction properties of the target test instruction set, fill the script generation template based on the target test instruction set, and generate a test script;

[0095] In the process of generating the test script, the syntax verification and logic check are performed on the test script to generate a test script that meets the requirements.

[0096] After determining the target test instruction set corresponding to the target application, the script is generated using the script generator. The role of the script generator is to use a rule engine to deepen the analysis of preliminary instructions to ensure that the instructions can be accurately converted into detailed test steps.

[0097] The script generator can identify and map preliminary instructions to specific test actions, such as interface operations, data validation, or interface calls, by using a set of pre-defined rule libraries; these rules need to understand the literal meaning of the instructions, capture the intent and context of the instructions, and then generate logically clear and coherent test scripts.

[0098] The script generator can also maintain a flexible script template library that supports multiple test frameworks and programming languages. According to the test intent and the nature of the instructions, select the appropriate template, fill in the target test instruction set according to the structure of the template, and generate a structured test script.

[0099] During the script generation process, the script generator will perform syntax verification and logic checking to ensure that the generated script is technically feasible and logically reasonable.

[0100] The test script generation method provided in the embodiment of the present application can be applied to software testing scenarios, game testing scenarios, Web testing scenarios, and mobile application testing scenarios. In the software testing scenario, the test script of the software application is automatically generated to improve the test coverage and efficiency; in the game testing scenario, the game test script is generated to automatically execute various test scenarios in the game; in the Web testing scenario, the test process of the Web application is automated, including interface testing and functional testing; in the mobile application testing scenario, a test script is generated for the mobile application to test the performance of the application on different devices and operating systems.

[0101] The following is an introduction to the test script generation method through an overall implementation process. Figure 2 As shown:

[0102] Step 201: Determine sample data for model training based on multimodal data corresponding to a plurality of applications, where the sample data includes a training data set and a validation data set.

[0103] Step 202: For the multimodal data in the training data set, for each application, based on the N test intentions associated with the application, determine N multimodal data in a set of multimodal data corresponding to the application, and for each multimodal data, generate an image feature vector, a video feature vector and a text feature vector, generate a comprehensive feature vector based on feature fusion, perform decision fusion based on the test decisions corresponding to each modal vector to generate a comprehensive test decision, and for each multimodal data, construct a training array based on the comprehensive feature vector and the comprehensive test decision.

[0104] Step 203: divide the multiple training arrays into a first array set and a second array set, pre-train the neural network model according to the first array set, and fine-tune the neural network model according to the second array set.

[0105] Step 204: Evaluate the model performance of the neural network model based on the validation data set, and determine the target model when the model performance of the neural network model meets the conditions.

[0106] Step 205: Input a test instruction generation request carrying multimodal data corresponding to the target application into the target model, and obtain a target test instruction set according to the processing result of the test instruction generation request by the target model.

[0107] Step 206: Generate a test script corresponding to the target application according to the target test instruction set.

[0108] In the above implementation process, a multimodal large model architecture is constructed, the target model is determined based on model training, the target model is used to determine the test instruction set corresponding to the application test, and the test script is generated according to the test instruction set. This can realize automatic generation of test scripts based on model training, improve the generation efficiency of test scripts as well as the accuracy and reliability of scripts.

[0109] The present application embodiment provides a test script generation device, such as Figure 3 As shown, including:

[0110] A determination module 301 is used to determine sample data participating in model training according to multimodal data corresponding to multiple applications, wherein the multimodal data includes image data representing the overall layout of the application, video data representing the user interaction situation, and voice data representing the application test intention;

[0111] The processing module 302 is used to train the pre-built neural network model according to the sample data, adjust the parameters and structure of the neural network model, and obtain a target model for outputting a test instruction set of the application;

[0112] An acquisition module 303 is used to process a test instruction generation request corresponding to a target application based on the target model, and acquire a target test instruction set corresponding to the target application;

[0113] The generating module 304 is used to generate a test script corresponding to the target application according to the target test instruction set.

[0114] Optionally, the sample data includes a training data set, and the processing module includes:

[0115] A processing and generating submodule, used for performing feature fusion processing and decision fusion processing on the multimodal data in the training data set to generate a plurality of training arrays including comprehensive feature vectors and matching comprehensive test decisions;

[0116] An adjustment submodule, for adjusting the parameters and structure of the neural network model based on a plurality of training arrays;

[0117] A determination submodule, used to determine the target model when the model performance of the neural network model meets the conditions;

[0118] Among them, the comprehensive feature vector includes image features for identifying user interface UI elements, video features for capturing user interactions, and voice features for understanding test intentions. Each comprehensive feature vector corresponds to a test intention, and the comprehensive test decision matching the comprehensive feature vector is adapted to the corresponding test intention.

[0119] Optionally, the processing generation submodule includes:

[0120] A first determining unit is configured to determine, for each application, N pieces of multimodal data from a set of multimodal data corresponding to the application based on N test intentions associated with the application, where N is an integer greater than or equal to 1;

[0121] A generating unit, configured to generate, for each piece of multimodal data, an image feature vector, a video feature vector and a text feature vector based on data encoding and standardization processing, wherein the text feature vector is determined based on speech data representing a test intention;

[0122] A fusion generation unit, configured to generate the comprehensive feature vector by performing feature fusion according to the image feature vector, the video feature vector and the text feature vector, and generate the comprehensive test decision by performing decision fusion based on the test decisions respectively corresponding to the image feature vector, the video feature vector and the text feature vector;

[0123] A generation unit is constructed to construct a training array for each multimodal data based on the corresponding comprehensive feature vector and the comprehensive test decision, so as to generate multiple training arrays for model training.

[0124] Optionally, the fusion generation unit is further used for:

[0125] Identify the test decision corresponding to the image feature vector based on the image decision model, identify the test decision corresponding to the video feature vector based on the video decision model, and identify the test decision corresponding to the text feature vector based on the speech decision model;

[0126] The test decisions of the three modes are fused to generate the comprehensive test decision.

[0127] Optionally, the adjustment submodule includes:

[0128] A pre-training unit, configured to pre-train the neural network model according to the first array set, so that the neural network model learns features of data of different modalities;

[0129] A fine-tuning unit, used for performing multiple model fine-tuning on the neural network model according to the second array set to adjust the parameters and structure of the neural network model;

[0130] The multiple training arrays are divided into the first array set and the second array set.

[0131] Optionally, the determining submodule includes:

[0132] An evaluation unit, configured to evaluate the model performance of the neural network model based on the validation data set;

[0133] A second determining unit, configured to determine that the neural network model is the target model in response to the model performance of the neural network model satisfying a condition;

[0134] The third determination unit is used to repeat the process of adjusting the parameters and structure of the neural network model based on the training data set and evaluating the model performance based on the verification data set in response to the model performance of the neural network model not satisfying the conditions, so as to determine the target model.

[0135] Optionally, the acquisition module is further used for:

[0136] In response to a test instruction generation request carrying multimodal data corresponding to the target application, inputting the test instruction generation request into the target model;

[0137] The target test instruction set is obtained according to the processing result of the target model on the test instruction generation request.

[0138] Optionally, the generating module is further used for:

[0139] Identify the test action set corresponding to the target test instruction set based on the rule base, and generate a test script according to the test action set; or

[0140] Determine a script generation template according to the test intent of the target application and the instruction properties of the target test instruction set, fill the script generation template based on the target test instruction set, and generate a test script;

[0141] In the test script generation process, the test script is subjected to syntax verification and logic checking to generate a test script that meets the requirements.

[0142] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0143] An embodiment of the present application also provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned test script generation method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0144] For example, Figure 4 FIG. 1 shows a schematic diagram of the physical structure of an electronic device. Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430, and the processor 410 is used to execute the various processes of the test script generation method of the embodiment of the present application, which will not be described one by one here.

[0145] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0146] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the above-mentioned test script generation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0147] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0148] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0149] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0150] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0151] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0152] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0153] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0154] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0155] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0156] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A test script generation method, characterized in that: include: Determine sample data for model training based on multimodal data corresponding to multiple applications, wherein the multimodal data includes image data representing the overall layout of the application, video data representing user interaction, and voice data representing application test intent; Training a pre-built neural network model according to the sample data, adjusting the parameters and structure of the neural network model, and obtaining a target model for outputting a test instruction set for an application; Processing a test instruction generation request corresponding to a target application based on the target model to obtain a target test instruction set corresponding to the target application; A test script corresponding to the target application is generated according to the target test instruction set.

2. The method according to claim 1, characterized in that The sample data includes a training data set, and the pre-built neural network model is trained according to the sample data, the parameters and structure of the neural network model are adjusted, and a target model for outputting a test instruction set of an application is obtained, including: Performing feature fusion processing and decision fusion processing on the multimodal data in the training data set to generate a plurality of training arrays including comprehensive feature vectors and matching comprehensive test decisions; Adjusting parameters and structure of the neural network model based on a plurality of training arrays; When the model performance of the neural network model meets the conditions, determining the target model; Among them, the comprehensive feature vector includes image features for identifying user interface UI elements, video features for capturing user interactions, and voice features for understanding test intentions. Each comprehensive feature vector corresponds to a test intention, and the comprehensive test decision matching the comprehensive feature vector is adapted to the corresponding test intention.

3. The method according to claim 2, characterized in that The performing of feature fusion processing and decision fusion processing on the multimodal data in the training data set to generate a plurality of training arrays including comprehensive feature vectors and matching comprehensive test decisions includes: For each application, based on N test intents associated with the application, determine N pieces of multimodal data from a set of multimodal data corresponding to the application, where N is an integer greater than or equal to 1; For each piece of multimodal data, based on data encoding and standardization processing, an image feature vector, a video feature vector and a text feature vector are generated, wherein the text feature vector is determined based on the speech data representing the test intention; Performing feature fusion according to the image feature vector, the video feature vector and the text feature vector to generate the comprehensive feature vector, and performing decision fusion based on the test decisions respectively corresponding to the image feature vector, the video feature vector and the text feature vector to generate the comprehensive test decision; For each piece of multimodal data, a training array is constructed based on the corresponding comprehensive feature vector and the comprehensive test decision to generate multiple training arrays for model training.

4. The method according to claim 3, characterized in that The generating the comprehensive test decision by performing decision fusion based on the test decisions corresponding to the image feature vector, the video feature vector and the text feature vector, respectively, comprises: Identify the test decision corresponding to the image feature vector based on the image decision model, identify the test decision corresponding to the video feature vector based on the video decision model, and identify the test decision corresponding to the text feature vector based on the speech decision model; The test decisions of the three modes are fused to generate the comprehensive test decision.

5. The method according to claim 2, characterized in that: The adjusting the parameters and structure of the neural network model based on multiple training arrays includes: Pre-training the neural network model according to the first array set so that the neural network model learns features of data of different modalities; Performing multiple model fine-tuning on the neural network model according to the second array set to adjust the parameters and structure of the neural network model; The multiple training arrays are divided into the first array set and the second array set.

6. The method according to claim 2 or 5, characterized in that: The sample data includes a validation data set, and when the model performance of the neural network model meets a condition, determining the target model includes: Evaluating the model performance of the neural network model based on the validation data set; In response to the model performance of the neural network model satisfying a condition, determining the neural network model as the target model; In response to the model performance of the neural network model not satisfying the conditions, the process of adjusting the parameters and structure of the neural network model based on the training data set and evaluating the model performance based on the validation data set is repeated until the model performance of the neural network model satisfies the conditions to determine the target model.

7. The method according to claim 1, characterized in that The step of processing a test instruction generation request corresponding to the target application based on the target model to obtain a target test instruction set corresponding to the target application includes: In response to a test instruction generation request carrying multimodal data corresponding to the target application, inputting the test instruction generation request into the target model; The target test instruction set is obtained according to the processing result of the target model on the test instruction generation request.

8. The method according to claim 1, characterized in that Generating a test script corresponding to the target application according to the target test instruction set includes: Identify the test action set corresponding to the target test instruction set based on the rule base, and generate a test script according to the test action set; or Determine a script generation template according to the test intent of the target application and the instruction properties of the target test instruction set, fill the script generation template based on the target test instruction set, and generate a test script; In the test script generation process, the test script is subjected to syntax verification and logic checking to generate a test script that meets the requirements.

9. A test script generating device, characterized in that: include: A determination module, used to determine sample data participating in model training according to multimodal data corresponding to a plurality of applications, wherein the multimodal data includes image data representing the overall layout of the application, video data representing the user interaction situation, and voice data representing the application test intention; A processing module, used to train a pre-built neural network model according to the sample data, adjust the parameters and structure of the neural network model, and obtain a target model for outputting a test instruction set for the application; An acquisition module, configured to process a test instruction generation request corresponding to a target application based on the target model, and acquire a target test instruction set corresponding to the target application; A generation module is used to generate a test script corresponding to the target application according to the target test instruction set.

10. An electronic device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the test script generating method according to any one of claims 1 to 8 when executed by the processor.

Images