Application software testing method and device

Abstract

The invention provides an application software testing method and device, and the method comprises the steps: determining a to-be-executed operation for a current graphical interface of application software, and if the to-be-executed type is a text input operation, determining a question statement corresponding to the to-be-executed operation according to the associated information of a to-be-triggered control corresponding to the to-be-executed operation, then, input content is determined for the to-be-executed operation according to the question statement based on the large language model, the to-be-executed operation is executed for the to-be-triggered control according to the input content, and therefore for application software with many text input controls, the input content can be generated through the large language model to serve as fuzzy test text input; therefore, the test of effective text input is realized, the path exploration depth of the fuzzy test is improved, and the comprehensiveness and effectiveness of the fuzzy test are improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a method and device for testing application software. Background Art

[0002] With the rapid development of software technology, in order to cope with the rapidly changing market demand and shorten the delivery cycle, the development iteration cycle of application software projects is often short. Software testing is the process of manual testing or automated testing to verify whether the software meets user needs. Manual testing is achieved through manual operation, and automated testing is achieved through automatic software operation. At present, the testing of application software is often highly dependent on manual execution, and it is difficult to perform full regression testing for each version iteration. In the process of manual testing, it is easy to have a fixed mindset and it is difficult to find defects caused by unreasonable operations. Due to the limitations of traditional testing, it is impossible to find all defects in application software.

[0003] Fuzz testing can be used as a supplement to traditional testing to improve the comprehensiveness of test coverage. Fuzz testing is a testing method that discovers software vulnerabilities by providing unexpected inputs to the target software and monitoring abnormal results. It also has problems such as poor effectiveness, high randomness, and low overall test efficiency, and cannot meet the needs of actual testing work. Summary of the invention

[0004] In view of this, the purpose of this application is to provide a testing method and device for application software, to improve the path exploration depth of fuzz testing, and to improve the comprehensiveness and effectiveness of fuzz testing. The specific scheme is as follows:

[0005] In a first aspect, the present application provides a method for testing application software, comprising:

[0006] Determine the operation to be performed for the current graphical interface of the application software;

[0007] If the operation to be performed is a text input operation, determining the question statement corresponding to the operation to be performed according to the associated information of the to-be-triggered control corresponding to the operation to be performed;

[0008] Determine input content for the operation to be performed according to the question sentence based on the large language model;

[0009] The operation to be performed is performed on the control to be triggered according to the input content.

[0010] Optionally, determining the operation to be performed for the current graphical interface of the application software includes:

[0011] If there is a target operation for the trigger operation corresponding to the operation control in the current graphical interface of the application software, and the target operation is a text input operation and the exploration state is not an explored state, then the operation to be executed is determined according to the target operation.

[0012] Optionally, the method further includes:

[0013] If, in the trigger operations corresponding to the operation controls in the current graphical interface, the exploration states of the text input operations are all explored states, and there are trigger operations whose exploration states are unexplored states, then the predicted operation information corresponding to the current graphical interface is determined by the human-computer interaction prediction model, and the predicted operation information is used to indicate the selection probability of the non-text input operation of the current graphical interface, and the distribution probability of the trigger position corresponding to the non-text input operation;

[0014] According to the predicted operation information, determining a non-text input operation that meets a preset condition from the current graphical interface as an operation to be executed, and determining a control to be triggered corresponding to the operation to be executed;

[0015] The operation to be executed is executed on the control to be triggered.

[0016] Optionally, determining the operation to be performed for the current graphical interface of the application software includes:

[0017] If the states of the trigger operations corresponding to the operation controls in the current graphical interface of the application software are all in the explored state, then according to the number of trigger operations whose exploration states in other graphical interfaces are not in the explored state, determine the target graphical interface from the other graphical interfaces;

[0018] An operation to be executed is determined according to a trigger operation corresponding to a next graphic interface of the current graphic interface in a path between the current graphic interface and the target graphic interface.

[0019] Optionally, the method further includes:

[0020] If the operation to be performed is a non-text input operation, the to-be-triggered control corresponding to the operation to be performed is determined according to the predicted operation information corresponding to the current graphical interface determined by the human-computer interaction prediction model; the predicted operation information is used to indicate the selection probability of the non-text input operation of the current graphical interface, and the distribution probability of the trigger position corresponding to the non-text input operation;

[0021] The operation to be executed is executed on the control to be triggered.

[0022] Optionally, the current graphical interface is displayed by performing a trigger operation on the jump graphical interface, and the method further includes:

[0023] The exploration state of the trigger operation between the current graphical interface and the jump graphical interface is updated to an explored state.

[0024] Optionally, the exploration state is represented by a graphical interface model, the graphical interface model includes a first node and a second node, the first node corresponds to the jump graphical interface, and the second node corresponds to the current graphical interface, and updating the exploration state of the trigger operation between the current graphical interface and the jump graphical interface to an explored state includes:

[0025] In the graphical interface model, a directed edge is added from the first node to the second node, and the directed edge corresponds to a trigger operation between the current graphical interface and the jump graphical interface.

[0026] Optionally, the associated information includes at least one of the attribute information of the control to be triggered, the attribute information of the associated control of the control to be triggered, and the attribute information of the current graphical interface;

[0027] If the operation to be performed is a text input operation, determining the question statement corresponding to the operation to be performed according to the association information of the to-be-triggered control corresponding to the operation to be performed includes:

[0028] If the operation to be performed is a text input operation, the associated information is reconstructed according to the sentence pattern of the prompt project to obtain a question sentence.

[0029] In a second aspect, the present application also provides a testing device for application software, including:

[0030] A fuzzy operation determination unit, used to determine an operation to be performed for the current graphical interface of the application software;

[0031] A question statement determining unit, configured to determine a question statement corresponding to the operation to be executed according to the association information of the to-be-executed control to be triggered corresponding to the operation to be executed, if the operation to be executed is a text input operation;

[0032] A text input content determination unit, configured to determine input content for the operation to be performed according to the question sentence based on a large language model;

[0033] The first operation execution unit is used to execute the operation to be executed on the control to be triggered according to the input content.

[0034] Optionally, the fuzzy operation determination unit is specifically used to:

[0035] If there is a target operation for the trigger operation corresponding to the operation control in the current graphical interface of the application software, and the target operation is a text input operation and the exploration state is not an explored state, then the operation to be executed is determined according to the target operation.

[0036] Optionally, the device further comprises:

[0037] An operation prediction unit, for determining predicted operation information corresponding to the current graphical interface through a human-computer interaction prediction model if, in the trigger operations corresponding to the operation controls in the current graphical interface, the exploration states of the text input operations are all in the explored state, and there are trigger operations whose exploration states are in the unexplored state, the predicted operation information is used to indicate the selection probability of the non-text input operation of the current graphical interface, and the distribution probability of the trigger position corresponding to the non-text input operation;

[0038] a fuzzy operation determination unit, configured to determine, based on the predicted operation information, a non-text input operation that meets a preset condition from the current graphical interface as an operation to be executed, and determine a control to be triggered corresponding to the operation to be executed;

[0039] The second operation execution unit is used to execute the operation to be executed on the control to be triggered.

[0040] Optionally, the fuzzy operation determination unit includes:

[0041] a target graphical interface determination unit, configured to determine the target graphical interface from other graphical interfaces according to the number of trigger operations whose exploration states are not the explored states in other graphical interfaces, if the states of the trigger operations corresponding to the operation controls in the current graphical interface of the application software are all in the explored state;

[0042] The operation determination subunit is used to determine the operation to be executed according to the trigger operation corresponding to the next graphic interface of the current graphic interface in the path between the current graphic interface and the target graphic interface.

[0043] Optionally, the device further comprises:

[0044] A non-text operation determination unit, for determining, if the operation to be performed is a non-text input operation, a to-be-triggered control corresponding to the operation to be performed according to predicted operation information corresponding to the current graphical interface determined by a human-computer interaction prediction model; the predicted operation information is used to indicate a selection probability of the non-text input operation of the current graphical interface, and a distribution probability of a trigger position corresponding to the non-text input operation;

[0045] The second operation execution unit is used to execute the operation to be executed on the control to be triggered.

[0046] Optionally, the current graphical interface is displayed by performing a trigger operation on a jump graphical interface, and the device further includes:

[0047] The exploration status updating unit is used to update the exploration status of the trigger operation between the current graphical interface and the jump graphical interface to an explored status.

[0048] Optionally, the exploration state is represented by a graphical interface model, the graphical interface model includes a first node and a second node, the first node corresponds to the jump graphical interface, the second node corresponds to the current graphical interface, and the exploration state updating unit is specifically used to:

[0049] In the graphical interface model, a directed edge is added from the first node to the second node, and the directed edge corresponds to a trigger operation between the current graphical interface and the jump graphical interface.

[0050] Optionally, the associated information includes at least one of the attribute information of the control to be triggered, the attribute information of the associated control of the control to be triggered, and the attribute information of the current graphical interface;

[0051] The question statement determination unit is specifically used for:

[0052] If the operation to be performed is a text input operation, the associated information is reconstructed according to the sentence pattern of the prompt project to obtain a question sentence.

[0053] The embodiment of the present application provides a testing method and device for application software, which determines an operation to be executed for the current graphical interface of the application software. If the type of operation to be executed is a text input operation, the question statement corresponding to the operation to be executed is determined according to the associated information of the control to be triggered corresponding to the operation to be executed, and then the input content for the operation to be executed is determined according to the question statement based on the large language model, and the operation to be executed is executed on the control to be triggered according to the input content. In this way, for application software with a large number of text input controls, the large language model can be used to generate input content as fuzzy test text input, thereby realizing the testing of valid text input, improving the path exploration depth of the fuzzy test, and improving the comprehensiveness and effectiveness of the fuzzy test. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1A schematic diagram of a process flow of a method for testing application software provided in an embodiment of the present application is shown;

[0056] Figure 2 A schematic diagram of an output text determination process provided by an embodiment of the present application is shown;

[0057] Figure 3 A schematic diagram of the use of an artificial interaction prediction model provided in an embodiment of the present application;

[0058] Figure 4 A schematic diagram of a non-text input operation prediction process in an embodiment of the present application;

[0059] Figure 5 A schematic diagram of a comprehensive testing method provided in an embodiment of the present application;

[0060] Figure 6 A schematic diagram of a specific flow chart of a method for testing application software provided in an embodiment of the present application;

[0061] Figure 7 A structural block diagram of a testing device for application software provided in an embodiment of the present application;

[0062] Figure 8 A structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below with reference to the accompanying drawings.

[0064] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0065] For ease of understanding, a method and device for testing application software provided in an embodiment of the present application are described in detail below with reference to the accompanying drawings.

[0066] refer to Figure 1 As shown, it is a flow chart of a method for testing application software provided in an embodiment of the present application, and the method may include the following steps.

[0067] S101, determining an operation to be executed in the current graphical interface of the application software.

[0068] In an embodiment of the present application, an application software (APP) may be tested. The application software may be a mobile banking application software. The mobile banking application software often requires text input to enter the next transaction link. Therefore, its graphical interface often has a large number of controls to be triggered corresponding to text input operations.

[0069] The current graphical interface (UI) of the application software is the currently displayed user graphical interface (GUI), which can also be called the current UI state. The user graphical interface is a computer operation user interface displayed in a graphical manner. The current graphical interface has UI elements, which are elements on the user interaction page, such as operation controls, pictures, etc. The operation controls can be triggered. During the testing of the application software, the controls to be triggered and the operations to be performed on the controls to be triggered can be determined, and the triggering operations can be performed on the controls to be triggered. Whether the functions of the application software are correct can be determined through the jump process of the graphical interface. The triggering operation can include click (touch), long press (long_touch), swipe up / down / left / right (swipe_up / swipe_down / swipe_left / swipe_right), return, text input (input_text), etc. Among them, other operations besides text input operations are non-text input operations.

[0070] In the embodiment of the present application, the exploration state of the jump operation between the graphical interfaces (i.e., the trigger operation of the operation control in the graphical interface) can also be recorded, and the exploration state includes an explored state and an unexplored state. Specifically, the current graphical interface is displayed by performing a trigger operation on the jump graphical interface, and the exploration state of the trigger operation between the current graphical interface and the jump graphical interface can be updated to an explored state. Specifically, the exploration state is represented by a graphical interface model (also referred to as a UI state transition diagram), and the graphical interface model includes a first node and a second node, the first node corresponds to the jump graphical interface, and the second node corresponds to the current graphical interface. Then, in the graphical interface model, a directed edge pointing from the first node to the second node can be added, and the directed edge corresponds to the trigger operation between the current graphical interface and the jump graphical interface, that is, when there is a directed edge between the first node and the second node, the exploration state of the trigger operation indicating the jump of the first node to the second node is an explored state, and when there is no directed edge between the first node and the second node, the exploration state of the trigger operation indicating the jump of the first node to the second node is an unexplored state.

[0071] In the case where the trigger operation has an exploration state, when it is determined that the trigger operation corresponding to the operation control in the current graphical interface of the application software has a target operation, and the target operation is a text input operation and the exploration state is not an explored state, the operation to be executed is determined according to the target operation. In other words, the operation to be executed can be determined from the unexplored text input operations, so that the unexplored text input operations can be explored, thereby improving the comprehensiveness of the test.

[0072] S102: If the operation to be executed is a text input operation, determine a question statement corresponding to the operation to be executed according to the association information of the to-be-triggered control corresponding to the operation to be executed.

[0073] If the operation to be performed is a text input operation, the input content needs to be determined, so as to automatically test the operation control corresponding to the text input operation. Specifically, the question statement corresponding to the operation to be performed can be determined according to the associated information of the to-be-triggered control corresponding to the operation to be performed.

[0074] The associated information of the control to be triggered may include at least one of the attribute information of the control to be triggered, the attribute information of the associated control of the control to be triggered, and the attribute information of the current graphical interface, so that the generation process of the question statement is a context-sensitive input prompt generation process. Figure 2 As shown, it is a schematic diagram of an output text determination process provided in an embodiment of the present application. The process in the dotted box is the process of generating a question statement. The current graphical interface is a UI interface with text input, and the associated information of the control to be triggered serves as the context information of the control to be triggered.

[0075] The attribute information of the control to be triggered may include prompt information related to the input content required for the control to be triggered (i.e., the text box), and the prompt information may be combined and stored in the string of the input component information for use, such as information related to the input content required for the text box in the prompt information, such as resource-id, text, label, content-desc, etc.; the associated control of the control to be triggered may be a control having a preset position relationship with the control to be triggered, such as a control on the same horizontal coordinate as the control to be triggered, or a control on the same vertical coordinate as the control to be triggered, and the attribute information of the associated control may be information such as text and resource-id, and the attribute information may be combined and stored in the string of the local context information for use; the attribute information of the current graphical interface may be such as the activity name of the current application, the APP name, and the number of text box controls on the current interface, and this information may be combined and stored in the string of the global context information.

[0076] The associated information can be determined by traversing the view hierarchy file of the current graphical interface and obtaining all the control information in the current graphical interface. The associated information can be obtained by combining valid information in at least one of the attribute information of the control to be triggered, the attribute information of the associated control of the control to be triggered, and the attribute information of the current graphical interface. Specifically, the valid information in the input component information, the local context information, and the global context information is combined, and the associated information obtained by the combination is used as the prompt information required for the input of the control to be triggered. Figure 2 shown.

[0077] After determining the associated information, the question sentence corresponding to the to-be-executed operation can be determined according to the associated information of the to-be-executed control corresponding to the to-be-executed operation. Specifically, the associated information can be reconstructed according to the sentence pattern of the prompt project to obtain a question sentence, and the question sentence can be input into the large language model as an input prompt of the large language model. The associated information can be reconstructed into a question sentence through a preset logic, and the preset logic can be automatically implemented by a computer program, that is, the question sentence can be automatically generated by inputting the associated information (equivalent to the input prompt), so that the question sentence can be input into the large language model.

[0078] Among them, prompt engineering refers to guiding the large language model to generate target results by designing and optimizing prompt words without updating model parameters. The large language model is a deep learning model trained based on massive text data. It can not only generate natural language text, but also deeply understand the meaning of the text and handle various natural language tasks such as text summarization, question answering, translation, etc.

[0079] In the process of constructing question statements through preset logic, an optimized question statement can be obtained based on the preset logic through the prompt engineering methodology of building structure, sorting, adding instructions, and making at least one of the presets. Therefore, without updating the model parameters, the large language model is guided to stably output valid and harmless text to ensure the validity of text input. This method can be adapted to common large language models.

[0080] First, the structure is to restrict the structure of the question. Specifically, in order to make the structure of the question as intuitive as possible, all the content is not put in one line, and appropriate line breaks are used to split the content structure of the question. The following is an example of the question:

[0081] Please generate 3 test input data for the input box according to the following interface content and prompt information.

[0082] Interface content: There is an input box on the interface of the mobile banking APP.

[0083] Tips: Please enter the detailed address.

[0084] Second, sorting refers to sorting the key information in the question. Generally speaking, if the key information is placed at the end, the model can output better results. For example, if the key information is "detailed address", the question sentence obtained is as follows:

[0085] Please generate 3 test input data for the input box according to the following interface content and prompt information.

[0086] Interface content: There is an input box on the interface of the mobile banking APP.

[0087] Tips: Please enter the detailed address.

[0088] Third, adding explanations is to emphasize and supplement details based on the associated information, such as detailed rules, output formats, etc. The following is an example of the question statement obtained:

[0089] Interface content: There is an input box on the interface of the mobile banking APP.

[0090] Tips: Please enter the detailed address.

[0091] The format of the test data is as follows:

[0092] 1.XXX

[0093] 2.XXX

[0094] Please directly input the test data and do not answer unnecessary questions.

[0095] Fourth, the purpose of making presets is to make presets for the scenarios, inform the model of the corresponding strategies, and effectively prevent the model from giving wrong answers. This is because the model sometimes makes up some wrong outputs due to the training data and text generation strategies. Making presets can avoid this problem. The following is an example of the question statement obtained:

[0096] You are a tester of a mobile banking app. Please generate test data that can be entered into the text box of the app.

[0097] Please generate 3 test input data for the input box according to the following interface content and prompt information.

[0098] Interface content: There is an input box on the interface of the mobile banking APP.

[0099] Tips: Please enter the detailed address.

[0100] The format of the test data is as follows:

[0101] 1.XXX

[0102] 2.XXX

[0103] Please directly input the test data and do not answer unnecessary questions.

[0104] Through the above method, question sentences that can accurately express needs and are suitable for a variety of large language models can be obtained, which is conducive to obtaining input content that is more in line with manual testing rules and makes the test more accurate.

[0105] S103: Determine input content for the operation to be performed according to the question sentence based on the large language model.

[0106] After determining the question sentence for the large language model, the input content for the operation to be executed can be determined based on the large language model according to the question sentence. The input content is the input text required by the control to be triggered corresponding to the operation to be executed, so as to use the large language model to generate the input content as the fuzzy test text input. Specifically, the question sentence can be input into the large language model, and the content output by the large language model is used as the input content of the operation to be executed. Figure 2 shown.

[0107] S104: Execute the operation to be executed on the control to be triggered according to the input content.

[0108] After determining the input content, the pending operation can be performed on the triggered control according to the input content. In this way, for application software with a large number of text input controls, a large language model can be used to generate input content as fuzzy test text input, thereby realizing the testing of valid text input, improving the path exploration depth of fuzz testing, and improving the comprehensiveness and effectiveness of fuzz testing.

[0109] After the pending operation is executed on the trigger control, an execution log of the pending operation can be formed, which can record the specific input time, input type, state transition and other information of the intelligent fuzzy test. The graphical interface model can also be updated according to the graphical interface displayed after the pending operation is executed.

[0110] The above description introduces the relevant introduction of text input operation.

[0111] In the embodiment of the present application, non-text input operations can also be performed on the current graphical interface. If the operation to be performed is a non-text input operation, the predicted operation information corresponding to the current graphical interface can be determined through the human-computer interaction prediction model, and then the to-be-triggered control corresponding to the to-be-executed operation is determined according to the predicted operation information, and then the to-be-executed operation is performed on the to-be-triggered control. Among them, the human-computer interaction prediction model can generate test inputs that simulate user operations based on the knowledge automatically learned from the artificially generated application interaction trajectory, so as to predict the next trigger operation for the current graphical interface, and the predicted operation information is used to indicate the selection probability of the non-text input operation of the current graphical interface, and the distribution probability of the trigger position corresponding to the non-text input operation.

[0112] The artificial interaction prediction model can be a deep learning model. Deep learning is a new research direction in the field of machine learning. It is introduced into machine learning to make it closer to the original goal - artificial intelligence. Deep learning is the inherent laws and representation levels of learning sample data. The information obtained in the learning process is of great help in the interpretation of data such as text, images and sounds. Its ultimate goal is to enable machines to have analytical learning capabilities like humans and to recognize data such as text, images and sounds. Deep learning has achieved many results in search technology, data mining, machine learning, machine translation, natural language processing, multimedia learning, speech, recommendation and personalization technology, and other related fields. Deep learning enables machines to imitate human activities such as seeing, hearing and thinking, solves many complex pattern recognition problems, and makes great progress in artificial intelligence related technologies.

[0113] Specifically, the artificial interaction prediction model can be a CNN-LSTM model, and the artificial interaction prediction model can also be called a CNN-LSTM model, or a human-computer interaction prediction model based on the CNN-LSTM model, or a CNN-LSTM model test generator, etc., wherein CNN (convolutional neural network) is a neural network model for image processing, which can automatically learn features in images and use these features for classification and recognition; LSTM (long short-term memory network) is a neural network model for sequence data processing, which is suitable for processing and predicting important events with very long intervals and delays in time series. Combining CNN and LSTM to obtain a CNN-LSTM model can be used to process image information or text information in sequence data. In practical applications, the CNN-LSTM model has been widely used in various fields, such as natural language processing, speech recognition, image recognition, and stock prediction.

[0114] The artificial interaction prediction model includes convolutional layer, LSTM layer and fully connected layer. The convolutional layer is one of the components of the convolutional neural network. Each convolutional layer is composed of several convolutional units. The parameters of each convolutional unit are optimized by the back propagation algorithm. The purpose of the convolution operation is to extract different features of the input. The first convolutional layer may only extract some low-level features such as edges, lines and corners. More layers of the network can iteratively extract more complex features from low-level features. The deconvolutional layer is one of the components of the convolutional neural network. The relationship with the convolutional layer is that the forward propagation process of the convolutional layer is the back propagation process of the deconvolutional layer, and the back propagation process of the convolutional layer is the forward propagation process of the deconvolutional layer. In image processing, given an input image, the pixels in a small area of ​​the input image are weighted averaged to become each corresponding pixel in the output image, where the weight is defined by a function, which is called the convolution kernel. The fully connected layer is one of the components of the convolutional neural network. Each node is connected to all nodes in the previous layer to integrate the features extracted previously. Long Short-Term Memory (LSTM) is a time recurrent neural network that is specially designed to solve the long-term dependency problem of general RNN (recurrent neural network). It is suitable for processing and predicting important events with very long intervals and delays in time series.

[0115] The artificial interaction prediction model can learn the pattern of user interaction with the application. The use of the artificial interaction prediction model goes through an offline stage of training the model using artificially generated interaction traces, and an online stage for guiding the generation of test inputs. Figure 3 , which is a schematic diagram of the use of an artificial interaction prediction model provided in an embodiment of the present application.

[0116] In the offline learning phase, a deep neural network model is used to learn the relationship between the GUI context and the user's executed interactions, and a human-computer interaction prediction model (also called a CNN-LSTM model or a CNN-LSTM model test generator) is obtained. The GUI context is represented as the visual information in the current UI state and the latest UI transition, while the interaction is represented as the operation type (touch, slide, etc.) and the location coordinates of the operation. By learning large-scale human interaction trajectories, the CNN-LSTM model can predict the probability distribution of the action type and action location of a new UI state, and the action location is used to determine the corresponding trigger control. The predicted distribution can then be used to calculate the probability of each UI element interacting with a human and how to interact with it.

[0117] Specifically, the human-computer interaction prediction model can be trained based on the public database RICO dataset, refer to Figure 3As shown. Specifically, financial application software data is screened from the public database RICO data set, and the user operation action sequence and state sequence are extracted according to the timestamp, UI state information and cursor position information in the data set to form a palm banking data set. Then, the human-computer interaction prediction model is trained based on the palm banking data set to learn the relationship between the current UI state of the application and the user operation. The human interaction prediction model includes a convolution layer, an LSTM layer and a fully connected layer. The training process can be specifically as follows: the palm banking data set is input into the convolution layer for feature extraction; the extracted features are used as the input of the LSTM layer to learn the sequence features; the model parameters are continuously updated using the back propagation algorithm; the output of the LSTM layer is converted into a probability distribution through the fully connected layer to obtain the probabilities of different user operation types and the coordinate positions of the user operations. Among them, the human-computer interaction prediction model can be trained in advance, and the palm banking data set can be used to fine-tune the human-computer interaction prediction model. Among them, the back propagation algorithm is an effective method for calculating gradients, which can quickly calculate the partial derivatives of each neuron in the neural network. Back propagation calculates the output of the network by first forward propagation, then back propagates the error from the output layer to the input layer, and finally calculates the partial derivative of each neuron based on the error. The core idea of ​​the backpropagation algorithm is to propagate the error backward through the chain rule and calculate the contribution of each neuron to the error.

[0118] The CNN-LSTM model test generator introduces the concept of UI context to simulate what people refer to when interacting with an application. i Indicates user operation (equivalent to the trigger operation of the trigger control), with S i Indicates the use of a i The UI state (i.e., graphical interface) corresponding to other timestamps captured before the timestamp of , a UI context content includes the current UI state S i and three latest UI transitions (S i-1 ,a i-1 )、(S i-2 ,a i-2 )、(S i-3 ,a i-3 ), the latest UI conversion can also be regarded as: the previous level view interface and its operation (S i-1 ,a i-1 ), the previous level view interface and its operations (S i-2 ,a i-2 ), as well as the previous level view interface of the previous level view interface and its operation (S i-3 ,a i-3). The current UI state (i.e., the current graphical interface) represents what the user sees when performing an operation, while the latest UI transition is used to model the user's potential intention during the current interaction session, that is, the next level view interface of the current view interface can be predicted based on the historical view interface of the current view interface.

[0119] The current UI state and the three latest UI transitions can be obtained by obtaining the application state through the input generator. The various operations (input methods) and probabilities generated by the CNN-LSTM model test generator can also be sent to the input generator so that the input generator can perform fuzzy test input on the application to implement the application test. Figure 3 shown.

[0120] In summary, the representation of UI context, i.e., the input of the CNN-LSTM model, can be a stack of images, including one 2-channel image for the current UI state and three 3-channel images for the three most recent UI transitions (each transition includes a 2-channel image for the UI state and a 1-channel image for the action). All images are scaled to 180×320 pixels. For ease of learning, a channel filled with 0s is added for the current UI state. Finally, the UI context is represented as a 4×180×320×3 vector.

[0121] The interactive model structure used in offline learning consists of a convolutional layer, a residual long short-term memory network module, and a deconvolutional layer (i.e., a fully connected layer). The convolutional layer uses two convolutional kernels, one is 3×3 with a RELU activation function, and the other is 2×2 without a RELU activation function. It accepts the current UI context as input and outputs the user operation a i The probability distribution of the position and operation type. Each neuron node in the neural network accepts the output value of the previous layer of neurons as the input value of the neuron, and passes the input value to the next layer. The input layer neuron node will directly pass the input attribute value to the next layer (hidden layer or output layer). In a multi-layer neural network, there is a functional relationship between the output of the upper layer node and the input of the lower layer node. This function is called an activation function. RELU is one of the commonly used activation functions.

[0122] In the online prediction stage, the non-text input prediction method uses a fine-tuned human-computer interaction prediction model to obtain the current UI state of the Palm Bank (i.e., mobile banking) APP and the three latest UI transitions through the input generator, and generates the type of human-like action (i.e., operation type) and the distribution probability of the corresponding target position. The operation to be performed can be determined based on the probability of the operation type. For example, the action type with the highest probability is obtained as the operation to be performed, that is, as the fuzzy test input, and the Palm Bank APP is operated through the input generator. At the same time, the current state of the Palm Bank APP is obtained in real time through the input generator, and the current UI state is recorded. Reference Figure 4 As shown, a schematic diagram of a non-text input operation prediction process provided in an embodiment of the present application is provided. In the human-computer interaction prediction model, the current UI state and the three latest UI states of the Palm Silver APP are obtained through the input generator, and the type of human-like action (i.e., operation type) and the distribution probability of the position are generated. After the input generator determines the fuzzy test input, the Palm Silver APP is operated.

[0123] In addition, the CNN-LSTM model test generator builds a GUI model called UTG (also known as UI transition graph model, UI state transition graph or graphical interface model) for the application under test. This is a directed graph whose nodes are UI states (i.e. graphical interfaces) and whose edges are operations that cause UI state transitions. UTG is constructed at runtime: each time the test generator observes a new state S i , it adds a new edge to the UTG i-1 ,a i-1 ,S i >, where S i-1 is the last observed UI state, a i-1 YesS i-1 Using UTG, the test generator can navigate to any known UI state by following the path of the state.

[0124] refer to Figure 5 As shown, it is a schematic diagram of a comprehensive testing method provided in an embodiment of the present application. The human-computer interaction prediction model is fine-tuned using the mobile banking data set. The human-computer interaction prediction model can determine the probability of non-text input operations. The input prompt can obtain a question sentence based on the prompt project. The question sentence is input into the big model to determine the input content of the text input operation. The input generator determines the fuzzy test input according to the input content of the text input operation or the probability of the non-text input operation, so as to test the application using the fuzzy test input.

[0125] ​In the embodiment of the present application, during the test process, the GUI model can be maintained to record the operations and corresponding UI state changes of the application program in the intelligent fuzzy testing method. For the fuzzy test text input and non-text input, according to the characteristics of the mobile banking application software, a biased random search algorithm with text input priority is adopted, and the final test input is determined based on the GUI model, prediction model, and UI control type. Specifically, a weighted selection is performed based on the characteristics of the UI element type of the mobile banking application software and whether the input has been explored to obtain the final input of the fuzzy test.

[0126] This is because the important transactions of the mobile banking system are transfer and remittance, which include multiple input processes, mostly text input, and all have non-empty checks. If the fuzzy test input is determined based on the priority predicted by the human-computer interaction prediction model, it is difficult to flow to each transaction process for testing. Therefore, it is necessary to optimize the fuzzy input algorithm to make a more effective combination of the inputs generated by the human-computer interaction prediction model and the text input prediction model to improve the effectiveness of fuzzy testing.

[0127] Specifically, refer to Figure 6 As shown, it is a specific flow chart of a testing method for application software provided in an embodiment of the present application. The method may also include S100, obtaining UI control information, the UI control information includes attribute information of the operation control in the current graphical interface, the attribute information includes the exploration status of the trigger operation corresponding to the operation control, the exploration status can be determined by a UI state transition diagram, all trigger operations that can be performed and the exploration status of these trigger operations are determined according to the attribute information of the operation control, and then the operation to be executed is determined based on the exploration status.

[0128] In the process of determining the operations to be executed, if there are unexplored text input operations in the triggering operations, the operations to be executed can be determined from the unexplored text input operations, for example, the first text input operation is used as the operation to be triggered; if there are no unexplored text input operations in the triggering operations, but there are unexplored non-text input operations, the operations to be triggered are determined according to the prediction results of the human-computer interaction prediction model, for example, the non-text input operation with the highest probability is used as the operation to be triggered; if all the triggering operations are explored, it is possible to jump to other graphical interfaces to continue testing. After determining the operations to be executed, the input generator can be used to inject the operations to be executed into the mobile banking application and update the exploration status. Repeat the aforementioned process of determining the operations to be executed and the process of executing the operations to be executed until all the triggering operations of the graphical interfaces are explored.

[0129] refer to Figure 6As shown, in the process of determining the operation to be triggered from the unexplored text input operation, S101 can be specifically as follows: S101A, when it is determined that the trigger operation corresponding to the operation control in the current graphical interface of the application software has a target operation, and the target operation is a text input operation and the exploration state is not an explored state, the operation to be executed is determined according to the target operation. Then S102 can be executed, that is, the operation to be executed can be determined from the unexplored text input operation, so that the unexplored text input operation can be explored to improve the comprehensiveness of the test.

[0130] refer to Figure 6 As shown, in the process of determining the operation to be triggered according to the prediction result of the human-computer interaction prediction model, the method may include S105, if the exploration state of the text input operation in the trigger operation corresponding to the operation control in the current graphical interface is all the explored state, and there is a trigger operation whose exploration state is the unexplored state, then the predicted operation information corresponding to the current graphical interface is determined by the human-computer interaction prediction model, and the predicted operation information is used to indicate the selection probability of the non-text input operation of the current graphical interface, and the distribution probability of the trigger position corresponding to the non-text input operation, and is used to determine the trigger control corresponding to the non-text input operation; S106, according to the predicted operation information, determine the non-text input operation that meets the preset conditions from the current graphical interface as the operation to be executed, and determine the control to be triggered corresponding to the operation to be executed; S107, execute the operation to be executed on the control to be triggered. Since the determined operation to be executed is a non-text input operation, it is not necessary to execute S102, S103 and S104.

[0131] refer to Figure 6 As shown, in the process of jumping to other graphical interfaces to continue testing, S101 can be specifically as follows: S101B, if the states of the trigger operations corresponding to the various operation controls in the current graphical interface of the application software are all explored states, then according to the number of trigger operations whose exploration states in other graphical interfaces are not explored states, the target graphical interface is determined from the other graphical interfaces; S101C, according to the trigger operation corresponding to the next graphical interface of the current graphical interface in the path between the current graphical interface and the target graphical interface, the operation to be executed is determined, so as to jump to the next graphical interface through the operation to be executed. After that, S102 is executed, that is, the state containing the most unexplored inputs and the shortest path required to jump from the current state to the state can be obtained according to the UI conversion record file, and the first action of the path is selected as the input, so as to switch the graphical interface and continue the test.

[0132] In addition, reference Figure 6As shown, after S101, the method may further include: S108, if the operation to be executed is a non-text input operation, determining the to-be-triggered control corresponding to the operation to be executed according to the predicted operation information corresponding to the current graphical interface determined by the human-computer interaction prediction model; the predicted operation information is used to indicate the selection probability of the non-text input operation of the current graphical interface, and the distribution probability of the trigger position corresponding to the non-text input operation; S107, executing the operation to be executed on the to-be-triggered control.

[0133] The embodiment of the present application introduces a method of deep learning model on the basis of fuzzy testing. Through the human-computer interaction prediction model, the large language model based on the prompt engineering, and the non-text input prediction method, the text input prediction method, and the biased random search algorithm with text input priority, the human-computer interaction prediction model can learn the human-computer interaction rules of financial applications, and can predict the action type of the user's next operation and the position distributed in the current UI state, forming a non-text input adapted to the intelligent fuzzy test of the mobile banking application. The text input is generated by the large language model based on the prompt engineering. By designing and optimizing the prompt words, the text input prediction model generates valid text input, which can generate valid text input and improve the path exploration depth of the fuzzy test. According to the characteristics of the UI element type of the mobile banking application software, a biased random search algorithm with text input priority adapted to the mobile banking application software is formulated, so that various fuzzy test inputs can be combined more efficiently, and the comprehensiveness and effectiveness of the fuzzy test are improved. In this way, the fuzzy test model has the ability to analyze and learn, and can realize the efficient combination of multiple fuzzy test inputs, thereby solving the bottleneck problems faced by traditional fuzzy testing in the field of financial technology in terms of effectiveness and comprehensiveness, and realizing the intelligent fuzzy test of the mobile banking application software.

[0134] Based on the above application software testing method, the present application embodiment also provides an application software testing device, referring to Figure 7 As shown, it is a structural block diagram of a testing device for application software provided in an embodiment of the present application, and the device may include:

[0135] A fuzzy operation determination unit 110, used to determine an operation to be performed for the current graphical interface of the application software;

[0136] The question statement determining unit 120 is used to determine the question statement corresponding to the operation to be executed according to the association information of the to-be-executed control to be triggered corresponding to the operation to be executed if the operation to be executed is a text input operation;

[0137] A text input content determination unit 130, configured to determine input content for the operation to be performed according to the question sentence based on a large language model;

[0138] The first operation execution unit 140 is configured to execute the operation to be executed on the control to be triggered according to the input content.

[0139] Optionally, the fuzzy operation determination unit is specifically used to:

[0140] If there is a target operation for the trigger operation corresponding to the operation control in the current graphical interface of the application software, and the target operation is a text input operation and the exploration state is not an explored state, then the operation to be executed is determined according to the target operation.

[0141] Optionally, the device further comprises:

[0142] An operation prediction unit, for determining predicted operation information corresponding to the current graphical interface through a human-computer interaction prediction model if, in the trigger operations corresponding to the operation controls in the current graphical interface, the exploration states of the text input operations are all in the explored state, and there are trigger operations whose exploration states are in the unexplored state, the predicted operation information is used to indicate the selection probability of the non-text input operation of the current graphical interface, and the distribution probability of the trigger position corresponding to the non-text input operation;

[0143] a fuzzy operation determination unit, configured to determine, based on the predicted operation information, a non-text input operation that meets a preset condition from the current graphical interface as an operation to be executed, and determine a control to be triggered corresponding to the operation to be executed;

[0144] The second operation execution unit is used to execute the operation to be executed on the control to be triggered.

[0145] Optionally, the fuzzy operation determination unit includes:

[0146] a target graphical interface determination unit, configured to determine the target graphical interface from other graphical interfaces according to the number of trigger operations whose exploration states are not the explored states in other graphical interfaces, if the states of the trigger operations corresponding to the operation controls in the current graphical interface of the application software are all in the explored state;

[0147] The operation determination subunit is used to determine the operation to be executed according to the trigger operation corresponding to the next graphic interface of the current graphic interface in the path between the current graphic interface and the target graphic interface.

[0148] Optionally, the device further comprises:

[0149] A non-text operation determination unit, for determining, if the operation to be performed is a non-text input operation, a to-be-triggered control corresponding to the operation to be performed according to predicted operation information corresponding to the current graphical interface determined by a human-computer interaction prediction model; the predicted operation information is used to indicate a selection probability of the non-text input operation of the current graphical interface, and a distribution probability of a trigger position corresponding to the non-text input operation;

[0150] The second operation execution unit is used to execute the operation to be executed on the control to be triggered.

[0151] Optionally, the current graphical interface is displayed by performing a trigger operation on a jump graphical interface, and the device further includes:

[0152] The exploration status updating unit is used to update the exploration status of the trigger operation between the current graphical interface and the jump graphical interface to an explored status.

[0153] Optionally, the exploration state is represented by a graphical interface model, the graphical interface model includes a first node and a second node, the first node corresponds to the jump graphical interface, the second node corresponds to the current graphical interface, and the exploration state updating unit is specifically used to:

[0154] In the graphical interface model, a directed edge is added from the first node to the second node, and the directed edge corresponds to a trigger operation between the current graphical interface and the jump graphical interface.

[0155] Optionally, the associated information includes at least one of the attribute information of the control to be triggered, the attribute information of the associated control of the control to be triggered, and the attribute information of the current graphical interface;

[0156] The question statement determination unit is specifically used for:

[0157] If the operation to be performed is a text input operation, the associated information is reconstructed according to the sentence pattern of the prompt project to obtain a question sentence.

[0158] An embodiment of the present application provides a testing device for application software, which determines an operation to be executed for the current graphical interface of the application software. If the type of operation to be executed is a text input operation, a question statement corresponding to the operation to be executed is determined based on the associated information of the control to be triggered corresponding to the operation to be executed, and then input content is determined for the operation to be executed based on the question statement based on a large language model, and the operation to be executed is executed on the control to be triggered based on the input content. In this way, for application software with a large number of text input controls, a large language model can be used to generate input content as fuzzy test text input, thereby realizing testing of valid text input, improving the path exploration depth of the fuzzy test, and improving the comprehensiveness and effectiveness of the fuzzy test.

[0159] In another aspect, the present application provides a computer device, see Figure 8 , which shows a structural diagram of a computer device provided in an embodiment of the present application, such as Figure 8 As shown, the device includes a processor 310 and a memory 320:

[0160] The memory 310 is used to store program codes and transmit the program codes to the processor;

[0161] The processor 320 is used to execute the test method of the application software provided in the above embodiment according to the instructions in the program code.

[0162] The computer device may include a terminal device or a server, and the aforementioned application software testing device may be configured in the computer device.

[0163] On the other hand, an embodiment of the present application further provides a storage medium, wherein the storage medium is used to store a computer program, and the computer program is used to execute the test method of the application software provided in the above embodiment.

[0164] In addition, an embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the test method for the application software provided in the above embodiment.

[0165] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by program instruction hardware, and the above program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the above storage medium can be at least one of the following media: read-only memory (English: Read-only Memory, abbreviated: ROM), RAM, magnetic disk or optical disk, etc. Various media that can store program codes.

[0166] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0167] The above is only a preferred implementation of the present application. Although the present application has been disclosed as a preferred embodiment, it is not intended to limit the present application. Any technician familiar with the art can use the above disclosed methods and technical contents to make many possible changes and modifications to the technical solution of the present application without departing from the scope of the technical solution of the present application, or modify it into an equivalent embodiment of equivalent changes. Therefore, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present application without departing from the content of the technical solution of the present application still falls within the scope of protection of the technical solution of the present application.

Claims

1. A method for testing application software, characterized in that: include: Determine the operation to be performed for the current graphical interface of the application software; If the operation to be performed is a text input operation, determining the question statement corresponding to the operation to be performed according to the associated information of the to-be-triggered control corresponding to the operation to be performed; Determine input content for the operation to be performed according to the question sentence based on the large language model; The operation to be performed is performed on the control to be triggered according to the input content.

2. The method according to claim 1, characterized in that Determining the operation to be performed for the current graphical interface of the application software includes: If there is a target operation for the trigger operation corresponding to the operation control in the current graphical interface of the application software, and the target operation is a text input operation and the exploration state is not an explored state, then the operation to be executed is determined according to the target operation.

3. The method according to claim 2, characterized in that The method further comprises: If, in the trigger operations corresponding to the operation controls in the current graphical interface, the exploration states of the text input operations are all explored states, and there are trigger operations whose exploration states are unexplored states, then the predicted operation information corresponding to the current graphical interface is determined by the human-computer interaction prediction model, and the predicted operation information is used to indicate the selection probability of the non-text input operation of the current graphical interface, and the distribution probability of the trigger position corresponding to the non-text input operation; According to the predicted operation information, determining a non-text input operation that meets a preset condition from the current graphical interface as an operation to be executed, and determining a control to be triggered corresponding to the operation to be executed; The operation to be executed is executed on the control to be triggered.

4. The method according to claim 1, characterized in that: Determining the operation to be performed for the current graphical interface of the application software includes: If the states of the trigger operations corresponding to the operation controls in the current graphical interface of the application software are all in the explored state, then according to the number of trigger operations whose exploration states in other graphical interfaces are not in the explored state, determine the target graphical interface from the other graphical interfaces; An operation to be executed is determined according to a trigger operation corresponding to a next graphic interface of the current graphic interface in a path between the current graphic interface and the target graphic interface.

5. The method according to claim 4, characterized in that The method further comprises: If the operation to be performed is a non-text input operation, the to-be-triggered control corresponding to the operation to be performed is determined according to the predicted operation information corresponding to the current graphical interface determined by the human-computer interaction prediction model; the predicted operation information is used to indicate the selection probability of the non-text input operation of the current graphical interface, and the distribution probability of the trigger position corresponding to the non-text input operation; The operation to be executed is executed on the control to be triggered.

6. The method according to any one of claims 1 to 5, characterized in that: The current graphical interface is displayed by performing a trigger operation on the jump graphical interface, and the method further includes: The exploration state of the trigger operation between the current graphical interface and the jump graphical interface is updated to an explored state.

7. The method according to claim 6, characterized in that The exploration state is represented by a graphical interface model, the graphical interface model includes a first node and a second node, the first node corresponds to the jump graphical interface, and the second node corresponds to the current graphical interface, and the updating of the exploration state of the trigger operation between the current graphical interface and the jump graphical interface to the explored state includes: In the graphical interface model, a directed edge is added from the first node to the second node, and the directed edge corresponds to a trigger operation between the current graphical interface and the jump graphical interface.

8. The method according to any one of claims 1 to 5, characterized in that: The associated information includes at least one of the attribute information of the to-be-triggered control, the attribute information of the associated control of the to-be-triggered control, and the attribute information of the current graphical interface; If the operation to be performed is a text input operation, determining the question statement corresponding to the operation to be performed according to the association information of the to-be-triggered control corresponding to the operation to be performed includes: If the operation to be performed is a text input operation, the associated information is reconstructed according to the sentence pattern of the prompt project to obtain a question sentence.

9. A testing device for application software, characterized in that: include: A fuzzy operation determination unit, used to determine an operation to be performed for the current graphical interface of the application software; A question statement determining unit, configured to determine a question statement corresponding to the operation to be executed according to the association information of the to-be-executed control to be triggered corresponding to the operation to be executed, if the operation to be executed is a text input operation; A text input content determination unit, configured to determine input content for the operation to be performed according to the question sentence based on a large language model; The first operation execution unit is used to execute the operation to be executed on the control to be triggered according to the input content.

10. The device according to claim 9, characterized in that The fuzzy operation determination unit is specifically used for: If there is a target operation for the trigger operation corresponding to the operation control in the current graphical interface of the application software, and the target operation is a text input operation and the exploration state is not an explored state, then the operation to be executed is determined according to the target operation.

Images