An automated testing method based on neural network

Abstract

In view of the problem of automated testing in the field of software testing, the present invention proposes an automated testing method based on a neural network. By classifying existing test videos and tracking the target of the cursor, a test software capable of automatically performing software testing can be written, and it has a self-optimization function to achieve the correct automated testing function. When an accidental interruption occurs during the testing process, it can also quickly resume to the previous associated test part for subsequent operations.

Description

Technical Field

[0001] The present invention belongs to the technical field of neural networks, and specifically relates to a device for realizing video classification and performing software automated testing through a CNN+RNN neural network. Background Art

[0002] In the current software testing field, automated testing has become the mainstream of system testing development. Now many domestic and foreign computer companies have developed their own automated testing software for product automated testing. However, the current automated testing software can only perform automated testing for certain types or several types of special products or functions. If other products or functions need to be tested, programmers need to modify the software. Especially when facing the correct design and testing of the same software system in different departments, due to different programming levels and design concepts in different departments, the design of the same software system will also be different. At this time, if relying on programmers to write test scripts or manual testing methods, the efficiency will be very low and the cost will be relatively high.

[0003] Video can be regarded as a frame sequence composed of a large number of images arranged in chronological order. For the classification of images, the current mainstream is to use a CNN convolutional neural network for classification. And because these images are arranged in chronological order, an RNN recurrent neural network can be added to extract the features of the time series. By using the CNN+RNN method, videos can be classified.

[0004] The invention patent CN201710928075.X, a software testing method, device, equipment and computer storage medium, discloses a software testing method. The software testing method includes the following steps: when a software testing request is detected, display a software testing task list according to the software testing request for testers to input test information; determine a test task according to the test information, and call test atoms in a preset test atom library according to the test task and configure them to form a test atom set; when an execution instruction for software testing is detected, call and execute the test atoms in the test atom set according to the execution parameters in the test information to complete the atomic operation of software testing. This invention also discloses a software testing device, equipment and computer storage medium. The purpose of this invention is to reduce the cost of software testing, improve the efficiency of software testing and avoid errors caused by manual operations by using atomic operations. This invention mainly still relies on manual testing, with low efficiency and high cost.

[0005] Currently, in the field of software testing, the testing of software interfaces and functions mainly relies on writing different test scripts for different test projects or directly conducting manual testing. With the increase in software functions or the addition of new software projects, the testing work becomes very cumbersome and difficult. In order to avoid reducing the work intensity of testers and improve testing efficiency, it is very necessary to design an automated testing method. As the core of deep learning, neural networks play an important role in the field of artificial intelligence. By constructing a neural network model, the computer can be made to complete some tasks that can only be done by humans. In the field of software testing, from the perspectives of video classification and object tracking, a test software and a neural network model can be written to make the test software track the test videos of existing test projects and automatically complete the software testing. Summary of the Invention

[0006] The present invention provides an automated testing method based on a neural network. The method includes:

[0007] Taking the video recorded in advance of the manual testing software demonstration as the deep learning training corpus to obtain the software interface to be opened and the software testing sequence each time; putting the recorded video into the neural network for training and obtaining the handle according to the opened interface; rewriting the underlying source code of SeleniumIDE to enable it to have the basic functions of the test software; obtaining the position parameters that the mouse needs to automatically click according to the video, and automatically inputting test statements according to the position where the mouse automatically clicks; when the information obtained by the input test statement is incorrect, searching for subsequent test statements for testing; when all test statements obtain incorrect information, changing the automatic click position and re-inputting; when the testing process is unexpectedly blocked, recording the content that has been tested, and in the next test, automatically testing the content that has been successfully tested, restoring to the previous associated test part, and continuing the subsequent testing; optimizing and updating the video classification model according to the comparison between the video test project tags and the test project names.

[0008] Further optionally, in the method as described above, the step of taking the video recorded in advance of the manual testing software demonstration as the deep learning training corpus to obtain the software interface to be opened and the software testing sequence each time includes:

[0009] Before testing, record the names of the test projects being carried out; for the test videos of the same software, store them in a folder named after the software with the naming of number plus name.

[0010] Further optionally, in the method as described above, the step of putting the recorded video into the neural network for training and obtaining the handle according to the opened interface includes:

[0011] Put the video into the trained neural network for downloading to obtain the frame images for subsequent operations; when the test software obtains incorrect information, based on the software interface handle, roll back the software interface to the previous state, as well as the software operation method for saving video elements.

[0012] Further optionally, in the method described above, by rewriting the underlying source code of SeleniumIDE to enable it to have the basic functions of test software, including:

[0013] By rewriting the underlying source code of SeleniumIDE, the test script of the software driver is rewritten to simulate mouse movement, single-click and double-click operations of the mouse, and text input operations on the keyboard.

[0014] Further optionally, in the method described above, according to the video, obtain the position parameters where the mouse needs to click automatically, and automatically input test statements according to the position where the mouse clicks automatically, including:

[0015] By converting the image color space to the HSV color space, adjusting the HSV threshold range to white to find the position of the mouse cursor; according to the position of the mouse cursor, sequentially simulate the basic functions of the test software automatically.

[0016] Further optionally, in the method described above, when the information obtained by the input test statement is incorrect, search for subsequent test statements for testing, including:

[0017] According to the found position of the mouse cursor, perform single-click and double-click operations of the mouse and text input operations on the keyboard in sequence; when recognizing text in the video, recognize it character by character and perform operations in sequence until the software interface is similar to the video picture.

[0018] Further optionally, in the method described above, when all test statements obtain incorrect information, change the automatic click position and re-enter, including:

[0019] When all test statements fail, it is determined that the positions of some buttons in the software in the video are inconsistent with the buttons of the current test software; by performing test statement operations on the buttons one by one until the software interface is similar to the video picture.

[0020] Further optionally, in the method described above, when the test process is accidentally blocked, record the content that has been tested. When testing next time, automatically test the content that has been successfully tested before, restore to the previous associated test part, and continue with the subsequent tests, including:

[0021] The content of recording what has been tested includes: mouse position, evaluation method, and test statements; through the recorded content, the machine can operate quickly and restore to the previous associated test part.

[0022] Further optionally, in the method as described above, optimizing and updating the video classification model according to the comparison between the video test item label and the test item name includes:

[0023] Comparing the video item label with the test item name, and retraining the videos with inconsistent labels and names through a neural network to obtain truly matching video classification labels.

[0024] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0025] The present invention addresses the problem of automated testing in the field of software testing, and proposes an automated testing method based on a neural network. By classifying existing test videos and tracking the target of the cursor, a test software that can automatically perform software testing can be written, and it has a self-optimization function to achieve correct automated testing functions. When an unexpected interruption occurs during the testing process, it can also quickly resume to the previous associated test part for subsequent operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a schematic flowchart of an automated testing method based on a neural network according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention;

[0028] Step 1: Record in advance the videos demonstrating the artificial test software as the deep learning training corpus, and obtain the software interface to be opened and the software test sequence for each test.

[0029] The requirements for recording the videos should be as follows: the specific software testing process; before performing a specific test, record the name of the test item being performed; for the test videos of the same software, store them in a folder named after the software with the number and name as the naming.

[0030] Using the screen recording software, after opening the software to be tested, perform different test items according to the test standard document and record the test videos. After the recording is completed, the recorded videos are stored in a folder named after the test item with the number name.

[0031] For example: Open the software to be tested, "Test.exe". After it opens and enters the initial interface, view the test standard document. According to the test standard document, clarify that the next test item to be carried out is to test whether the user login function of the software is normal, and use the screen recording function built into the Windows system to start screen recording. After the test item is completed, stop screen recording, and save the recorded video in the format of "01.mp4" to the folder named "User Login".

[0032] In step 2, use the video recorded in step 1 as the input, record the name of the test item carried out as the label, and construct a neural network model combining CNN and RNN for training.

[0033] First, construct a neural network model. Download the pre-trained InceptionConvNet CNN neural network model and use it directly. The GoogleNet model will traverse all frames of the video, input the output of the last pooling layer of the network into the RNN recurrent neural network model of a single-layer LSTM to obtain a video classification model. Use the videos of different test items stored in step 1 as the input, and the name of the folder where they are located as the label. The label will be automatically encoded to train the video classification model.

[0034] For example: After constructing the video classification model, divide the total of 40 test videos in the "User Login" and "Draw Graphics" folders obtained in step 1 into a training set and a test set. Among them, there are 30 in the training set and 10 in the test set. Input the videos in the training set into the video classification model. After training for 50 epochs, it is found that the accuracy rate remains at 93.34% and no longer increases. At this time, stop training. Input the videos in the test set and find that the accuracy rate is 90.48%, which can be used for normal use.

[0035] Step 3: Download SeleniumIDE and rewrite the underlying source code to write the basic functions of the test software, so that it has the basic functions required for automatic software testing.

[0036] Download SeleniumIDE. Since Selenium is originally a browser-driven software, its underlying source code needs to be rewritten to implement the rewriting of software drivers and test scripts, and to implement the basic functions of software testing, including opening software programs in the Windos system, keyword search in the software interface, simulation functions of the mouse and keyboard, obtaining the handle of the software interface, screen capture and recording functions.

[0037] For example: After rewriting the underlying source code of SeleniumIDE and running it, it can automatically open the software to be tested, "Test.exe", and simulate the functions of the mouse and keyboard in the software. When the software interface switches, it can capture the handles of different interfaces and switch the software interface according to the handles.

[0038] Step 4: Write the test algorithm part for the test software in Step 3.

[0039] Its test algorithm includes the following process: Obtain the content of the test standard document, find the corresponding classified video in Step 4 according to the test item name in the document, download the pre-trained neural network model, and obtain the image frames in the video. At the same time, use the screen capture function of the test software to capture the current interface in real time, and compare the similarity of the software interface after opening the software program with the initial software interface in the video using the perceptual hashing algorithm. If they are not similar, use the simulated mouse and keyboard functions to open the next or previous level interface of this interface for comparison until it is the same as the initial software interface in the video.

[0040] The steps of the perceptual hashing algorithm include: Resize the image to an 8x8 size; Convert the resized image to 64-level grayscale; Calculate the converted DCT (Discrete Cosine Transform) value, which decomposes the image into frequency clusters and ladders. At this time, the DCT value is a 32*32 matrix; Only keep the 8*8 matrix in the upper left corner of the DCT; Calculate the average value of all 64 values in the matrix; Set the hash value of 0 or 1 according to the DCT average value, set those greater than or equal to the DCT average value to "1", and those less than the DCT average value to "0"; Combine the comparison results of the previous step to form a 64-bit integer, and the combination order must be the same, that is, from left to right and from top to bottom; Compare these 64-bit integers. If the number of different data bits does not exceed 5, it means that the two images are very similar.

[0041] Step 5: According to the recorded video, predict the position where the mouse needs to be automatically clicked according to the algorithm described in Step 4, and automatically input the test statement.

[0042] Take the recorded video as a parameter and input it into the trained neural network model. The GoogleNet network will traverse the video frames one by one, and at the same time use the perceptual hashing algorithm to compare the similarity. If the determination result of the current frame image and the previous frame image is not similar, that is, there is a large change. Then convert the image color space of the previous frame image to the HSV color space, adjust the HSV threshold range to white, and find the position of the mouse cursor. After rewriting the SeleniumIDE source code, the test software will move the simulated mouse to the same position, and simulate single-click left button, double-click or keyboard input respectively. If the changed software interface is similar, find the position where the mouse needs to be automatically clicked.

[0043] Further, if the positions of the icons or buttons for video and software testing change, that is, after operating according to the mouse cursor position in the video, the transformed interface image is not similar to the picture in the video. Then, by simulating the mouse function, click each button in the software one by one, temporarily save the mouse position, and compare the similarity of the transformed pictures until a similar image is found. The temporarily saved mouse position is the position that needs to be automatically clicked.

[0044] For example: After the test software opens "test.exe", the test software will obtain the content of the test standard document, find the test item with the test item name of "user login" according to the test item name in the document, and find its corresponding video according to the test item name. Put the video into the neural network to obtain each frame of the picture. When the video content is a mouse click on a button and then the interface changes, obtain the previous frame picture at this time. Adjust the HSV threshold range, find the mouse position, and in the test software, according to the mouse simulation function in step three, perform test statements at the same position. Each operation will perform a hash algorithm comparison between the video picture and the test software interface until it is determined to be very similar.

[0045] Step six, when the information obtained from the input test statement is incorrect, search for subsequent test statements for testing. When all test statements obtain incorrect information, change the automatic click position and re-enter.

[0046] According to the GoogleNet deep learning network, obtain each frame of the picture in the video. Based on the obtained mouse click position, perform operations such as single-click, double-click of the mouse or keyboard input. When the software interface changes and is similar to the picture presented in the video, continue the operation. If not, it is necessary to roll back to the previous interface and test the subsequent evaluation statements.

[0047] Further, if the evaluation statement is a double-click of the mouse, try a double-click of the mouse after rolling back.

[0048] Further, when the information obtained from both single-click and double-click of the mouse is incorrect, that is, the evaluation statement is keyboard input, mark the frame picture at the start of the input statement and the frame picture after the input is completed (i.e., the previous frame picture with a large interface change). Within the marked range, start from the first frame and perform a perceptual hash algorithm comparison with all subsequent frame pictures in turn. Set a threshold for the comparison of 64-bit integers in the DCT so that the loop stops when there is a difference of only one character between two frame pictures, and make a continue label. Identify the text in the input box currently, enter the same text in the evaluation software. If the interface is inconsistent with the picture after the label segment, start from the continue label and perform a hash comparison with the subsequent frame pictures in turn. The steps are the same as above.

[0049] Further, if all test statements obtain incorrect information, that is, it is determined that the positions of the icons or buttons in the video and software test have changed, then according to the special situation mentioned in Step Five, perform the operation of clicking each button in the software one by one until the mouse position is finally found, and then re-enter the test statement.

[0050] The advantage of doing this is that when the video evaluation statement is to input text and a lot of text is input. This approach can reduce the error of simultaneously recognizing a large number of texts, and at the same time, it can also determine the same picture when relatively short keyword phrases are input.

[0051] For example: The video shooting duration is too long, so that the positions of some buttons in the software and the buttons in the video have changed, and at this time, the test item requires entering a long string of text in the text box. Find the position of the text box that appears in the video according to the steps described above, and synchronize it in the test software. Because the positions are inconsistent, when testing the software, all the information obtained by the test statements fails. Each button in the software will be clicked one by one, and operations such as single-clicking, double-clicking the mouse, and entering text on the keyboard will be performed in sequence (when entering text, it will search character by character). When searching for the first five texts in a button at a random position, the video screen is very similar to the software interface, and proceed to the next step.

[0052] Step Seven, when the test process is unexpectedly blocked, record the content that has been tested. When testing next time, automatically test the content that was successfully tested before, restore to the previous associated test part, and continue with the subsequent tests.

[0053] When the software interface changes and is very similar to the video screen after the test statement, automatically obtain the video elements in the video. The video elements include the picture image of the current frame, the video progress bar number, the mouse click position, the evaluation method, and the statement. When the test process is unexpectedly blocked, sort the video progress bar numbers from small to large, and then quickly obtain the mouse position, evaluation method, and statement of the video elements in sequence, and perform automatic operations to restore to the associated test part.

[0054] Further, because it is only saved when it changes after the operation and the interface is similar to the video, the saved video elements are available, and the associated test part can be quickly restored according to the saved video elements.

[0055] Further, obtain the automatic mouse click position according to the above method and save the position parameters.

[0056] Further, the evaluation methods include single-clicking, double-clicking the mouse, and entering text on the keyboard, which are saved as 1, 2, and 3 respectively.

[0057] Further, when the operation is successful, save the text entered on the keyboard.

[0058] For example: When testing the software up to step five, when the button positions are inconsistent, clicking each button one by one, recognizing the text, and testing is a step that takes a relatively long time. If the operation is successful, the video elements, i.e., the new mouse positions and the content of the input text, are automatically saved. When the testing process is accidentally interrupted, according to the numbers sorted on the progress bar, the operations are quickly performed in sequence based on the saved video elements, reducing the process of re-traversing and extremely shortening the time. At the same time, for users, they can quickly resume in case of accidental interruption, greatly enhancing the user experience.

[0059] Step Eight: Optimize and update the video classification model.

[0060] Run the testing software to obtain the testing video. Use the video classification model obtained in step two to classify the testing video to get the label of the testing item to which this video belongs, and compare it with the testing item name in step four. If they are different, then use this video as the input and the testing item name in step four as the label to retrain this neural network model.

[0061] For example: After the testing software tests the software for the testing item of "user login" and obtains the screen recording video, at this time, input the video into the video classification model obtained in step two for classification, and it is found that this video belongs to the testing item of "person name query", which does not match the actual situation. So at this time, use this video as the input and "user login" as the label to input into the video classification model for retraining.

[0062] The program for implementing the information control of the present invention can be written in one or more programming languages or combinations thereof to write the computer program code for performing the operations of the present invention. The programming languages include object-oriented programming languages - such as Java, python, C++, and also include conventional procedural programming languages - such as the C language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network - including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0063] In several embodiments provided by the present invention, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation.

[0064] The unit described as a separation component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or may be distributed over multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0065] In addition, each functional unit in various embodiments of the present invention may be integrated in a processing unit, may exist separately as individual physical units, or two or more units may be integrated in one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a hardware plus software functional unit. The above-mentioned integrated unit implemented in the form of a software functional unit may be stored in a computer-readable storage medium. The above-mentioned software functional unit stored in a storage medium includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to execute some steps of the methods described in various embodiments of the present invention.

[0066] The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc that can store program codes.

[0067] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. An automated testing method based on a neural network, characterized in that, The method includes: Taking the video recorded in advance of the manual test software demonstration as the deep learning training corpus, and obtaining the software interface to be opened and the software test sequence for each test; Putting the recorded video into the neural network for training, and obtaining the handle according to the opened interface; Rewriting the underlying source code of SeleniumIDE to enable it to have the basic functions of testing software; The test software writes the test algorithm part, including: obtaining the content of the test standard document, finding the corresponding classified video according to the test item name in the document, and obtaining the image frames in the video; at the same time, using the screen capture function of the test software to capture the current interface in real time; Obtaining the position parameters that the mouse needs to automatically click according to the video, and automatically inputting the test statement according to the position where the mouse automatically clicks, including: taking the recorded video as a parameter and inputting it into the trained neural network model. The GoogleNet network will traverse the video frame by frame, and at the same time, use the perceptual hashing algorithm to compare the similarity. If the determination result of the current frame image and the previous frame image is dissimilar, then convert the image color space of the previous frame to the HSV color space, adjust the HSV threshold range to white, find the position of the mouse cursor. After the test software rewrites the SeleniumIDE source code, it will simulate the mouse moving to the same position, and respectively simulate single-clicking the left button, double-clicking or keyboard input. If the changed software interface is similar, then find the position where the mouse needs to automatically click; When the information obtained by the input test statement is incorrect, search for subsequent test statements for testing; when all test statements obtain incorrect information, change the automatic click position and re-enter; When the test process is unexpectedly blocked, record the content that has been tested. At the next test, automatically test the content that has been successfully tested, restore to the previous associated test part, and continue with the subsequent test; Optimizing and updating the video classification model according to the comparison between the video test project label and the test project name; 2. The method according to claim 1, wherein The step of taking the video recorded in advance of the manual test software demonstration as the deep learning training corpus, and obtaining the software interface to be opened and the software test sequence for each test includes: Before the test, record the name of the test project to be carried out; for the test videos of the same software, store them in the folder named after the software with the number and name as the name.

3. The method according to claim 1, wherein, The step of putting the recorded video into the neural network for training and obtaining the handle according to the opened interface includes: Putting the video into the downloaded and trained neural network to obtain the frame image of each frame for subsequent operations; when the test software obtains incorrect information, according to the software interface handle, roll back the software interface to the interface before the change, and the software operation method for saving video elements.

4. The method according to claim 1, wherein, The step of rewriting the underlying source code of SeleniumIDE to enable it to have the basic functions of testing software includes: By rewriting the underlying source code of SeleniumIDE, realizing the rewriting of the software-driven test script, and having the operations of simulating mouse movement, single-clicking and double-clicking the mouse, and inputting text with the keyboard.

5. The method according to claim 1, wherein, Obtaining the position parameters where the mouse needs to automatically click according to the video, and automatically inputting test statements according to the positions where the mouse automatically clicks, including: Converting the image color space to the HSV color space, adjusting the HSV threshold range to white, and finding the position of the mouse cursor; according to the position of the mouse cursor, automatically simulating the basic functions of the test software in sequence.

6. The method according to claim 1, wherein When the information obtained from the input test statements is incorrect, searching for subsequent test statements for testing, including: Performing operations of single-clicking and double-clicking the mouse and inputting text on the keyboard in sequence according to the found position of the mouse cursor; when recognizing text in the video, recognizing each character and performing operations in sequence until the software interface is similar to the video picture.

7. The method according to claim 1, wherein When all test statements obtain incorrect information, changing the automatic click position and re-inputting, including: When all test statements fail, determining that the positions of some buttons in the video and the buttons in the current test software are inconsistent; performing operations on the buttons with test statements one by one until the software interface is similar to the video picture.

8. The method according to claim 1, wherein When the test process is accidentally blocked, recording the content that has been tested. During the next test, automatically testing the content that was successfully tested before, restoring to the previous associated test part, and continuing with the subsequent tests, including: The content that has been tested includes: the mouse position, the evaluation method, and the test statements; restoring to the previous associated test part according to the recorded content.

9. The method according to claim 1, wherein, Optimizing and updating the video classification model according to the comparison between the video test item tags and the test item names, including: Comparing the video item tags with the test item names, and retraining the videos with inconsistent tags and names through a neural network to obtain the truly matching video classification tags.

Images