Automatic webpage testing method based on deep learning

Abstract

The invention relates to an automatic webpage testing method based on deep learning, which comprises the following steps of: accessing a to-be-tested webpage by using a browser automation tool, acquiring image data of the webpage, and preprocessing the intercepted webpage image data; constructing a text recognition model, inputting the preprocessed webpage image data into the verified text recognition model, and outputting text content; constructing a text line positioning model, inputting the output text content into the text line positioning model, and identifying position information of each line of text in the image; constructing a process function detection model, and identifying each functional element in the webpage based on the verified process function detection model; and according to output results of the text recognition model and the process function detection model, compiling an automatic test script, performing automatic test on the webpage, and generating an automatic test report. The testing accuracy and efficiency are improved, manual intervention is reduced, the software development period is shortened, and the product quality is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of web page testing methods, and in particular relates to a web page automated testing method based on deep learning. Background Art

[0002] Before a web page is put into use, it needs to be tested to ensure that all users can browse the web page and use the services provided by the website normally after the website is released. The realization of this goal depends on comprehensive testing of the website to ensure its quality and user experience. First of all, since the web page involves multiple technologies, the information structure design and the correctness of the file organization structure of the entire website need to be tested to ensure the normal operation of the website functions. Due to the technical differences between different browsers, cross-browser testing of web pages is required to ensure that the same web page displays the same effect in different browsers. Considering that different network operating systems may produce different effects, web pages need to be tested under different operating systems to ensure that the performance of web pages is consistent in different environments. Test the actual effect of the web page under environmental conditions lower than the assumed environment to ensure that the web page can work normally under various network conditions. Test the navigation design of the website to ensure its stability and correctness to avoid errors or inconveniences encountered by users during use.

[0003] At present, the commonly used web page automation testing method is to use the selenium automation testing tool to test the web page. For the specific testing process, please refer to the attached Figure 2 As shown, Selenium automated testing has limitations and shortcomings in scenarios where the web page structure changes frequently, performance requirements are high, and there is a lot of demand for dynamic content processing. It is sensitive to structural changes of web pages, has slow performance, poor cross-browser compatibility, has difficulty processing dynamic content, is not suitable for image content, and cannot process verification codes.

[0004] Therefore, there is an urgent need to improve the existing web page automation testing methods to adapt to scenarios such as frequent changes in web page structure, high performance requirements, and more demands for dynamic content processing, and to solve technical problems such as sensitivity to structural changes of web pages, slow performance, poor cross-browser compatibility, difficulty in processing dynamic content, not suitable for image content, and inability to process verification codes. Summary of the invention

[0005] The purpose of the present invention is to provide a web page automation testing method based on deep learning, which is used to adapt to scenarios such as frequent changes in web page structure, high performance requirements, and more demands for dynamic content processing, and to solve the technical problems of being sensitive to structural changes of web pages, slow performance, poor cross-browser compatibility, difficulty in processing dynamic content, not suitable for image content, and inability to process verification codes.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0007] A web page automated testing method based on deep learning, comprising the following steps:

[0008] S1: Use a browser automation tool to access the web page to be tested, and use a screenshot tool to take a screenshot of the entire page, obtain image data of the web page, and pre-process the captured web page image data;

[0009] S2: construct a text recognition model, and train, test and verify the constructed text recognition model, input the preprocessed web page image data into the verified text recognition model, and output text content;

[0010] S3: constructing a text line positioning model, and training, testing and verifying the constructed text line positioning model, inputting the output text content into the text line positioning model, and identifying the position information of each line of text in the image;

[0011] S4: construct a process function detection model, and train, test and verify the constructed process function detection model, and identify various functional elements in the web page based on the verified process function detection model;

[0012] S5: Based on the output results of the text recognition model and the process function detection model, write automated testing scripts, perform automated testing on web pages, and generate automated testing reports.

[0013] Preferably, the specific process of preprocessing the intercepted web page image data in step S1 is as follows:

[0014] S11: performing denoising processing on the intercepted web page image data to delete invalid image data and noise data therein;

[0015] S12: performing geometric transformation processing on the web page image data after the denoising process to adjust the geometric properties of the web page image to consistency, including rotating, scaling, and translating the image data;

[0016] S13: Use contrast stretching, dynamic range compression, and grayscale slicing to enhance the image and highlight important image features in web images.

[0017] Preferably, after step S13, the image data is grayed out, and the color of each pixel in the web page image is represented by a gray value, the gray value range is 0-255, and the R, G, and B components in the color image are respectively used as the gray values ​​of the three gray images, or the maximum value of the three components is taken as the gray value, or the average value of the three components is calculated as the gray value.

[0018] Preferably, the specific process in step S2 is as follows:

[0019] S21: Obtain historical web page image data, annotate the historical web page image data, and construct a historical web page image data set, dividing the annotated web page image data into a training set, a test set, and a verification set;

[0020] S22: inputting the training set into the text recognition model to train it;

[0021] S23: input the test set into the text recognition model to test it, and output the test result, and determine whether the test result meets the preset test requirements. If so, execute step S24; if not, re-input a new training set for training;

[0022] S24: input the verification set into the text recognition model to verify its test results, and output the verification results to determine whether the verification results meet the preset verification requirements. If so, execute step S3; if not, re-input a new training set for training;

[0023] The process of training, testing and verifying the text line positioning model and the process function detection model in step S3 and step S4 is the same as the process of training, testing and verifying in the specific process of step S2.

[0024] Preferably, before training, testing and verifying the text recognition model, the text line positioning model and the process function detection model, the corresponding models are optimized, and the model optimization process is as follows:

[0025] Initialize the parameters of the text recognition model, text line location model, and process function detection model: including setting weights and biases, and initializing the first-order moment estimation variable m and the second-order moment estimation variable v, both of which have the same dimensions as the model parameters;

[0026] Compute sample gradients for text recognition models, text line localization models, and process function detection models;

[0027] The first-order moment estimation variable m and the second-order moment estimation variable v are updated: the first-order moment estimation of the current gradient, i.e., the average gradient, is calculated, m=β1m+(1-β1)g, and the second-order moment estimation variable v=β2v+(1-β2)g2, wherein β1 and β2 are adjustable exponential decay rates, and the values ​​are 0.9 and 0.999, respectively;

[0028] The first-order moment estimation variable m and the second-order moment estimation variable v are corrected: correcting the first-order moment estimation variable m, The second-order moment estimate variable v is corrected: Where t represents the current iteration number;

[0029] Update the model parameters of the text recognition model, text line location model, and process function detection model: estimate the variables based on the corrected first-order moment and the second moment estimation variable And the learning rate α updates the model parameters, θ represents the parameters of the model, ∈ is a very small number, take 10 -8 .

[0030] Preferably, the functional elements in step S4 include titles, buttons, input boxes, links, forms, navigation bars, block elements, scripts, databases, programs, and footer blocks.

[0031] Preferably, the content of the automated test report in step S5 includes the test environment, test objectives, test cases, execution status, defect list, and performance test results.

[0032] The beneficial effects of the present invention include:

[0033] The web page automation testing method based on deep learning provided by the present invention uses a browser automation tool to access the web page to be tested, obtains the image data of the web page, and pre-processes the intercepted web page image data; builds a text recognition model, and inputs the pre-processed web page image data into the verified text recognition model, and outputs the text content; builds a text line positioning model, inputs the output text content into the text line positioning model, and identifies the position information of each line of text in the image; builds a process function detection model, and identifies each functional element in the web page based on the verified process function detection model; writes an automated test script according to the output results of the text recognition model and the process function detection model, performs automated testing on the web page, and generates an automated test report.

[0034] First, the web page interface is understood through image parsing, which is more flexible than the DOM parsing of Selenium in the prior art. Because it does not rely on the web page source code or DOM structure, it can handle various types and styles of web pages without being affected by changes in the web page structure.

[0035] It can process complex contents in web page image data, such as charts, animations, etc., while Selenium in the prior art mainly targets the DOM structure of web pages and may not be able to effectively process these image contents.

[0036] Using deep learning methods for image parsing does not require access to the source code of the web page, which means that testing can be performed on dynamically generated web pages or when the source code of the web page is inaccessible. Selenium needs to directly operate the web page DOM, which may not be possible in this case.

[0037] By implementing automated control of web page elements through image analysis, web page elements can be located and operated more accurately, while Selenium implements automated control through the operation of DOM elements, which may be affected by changes in the web page structure and lead to inaccurate positioning. Deep learning methods are not restricted by language and can process web page content in multiple languages, while Selenium may need to be adapted for web pages in different languages. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the process of the deep learning-based web page automation testing method of the present invention.

[0039] Figure 2 The figure is a schematic diagram of the Selenium automated testing process in the prior art.

[0040] Figure 3 A schematic diagram of a captured web page image according to the present invention. DETAILED DESCRIPTION

[0041] The following is combined with Figure 1 to Figure 3 The present invention is further described in detail:

[0042] Example 1

[0043] See attached Figure 1 As shown, a web page automated testing method based on deep learning includes the following steps:

[0044] S1: Use a browser automation tool to access the web page to be tested, and use a screenshot tool to take a screenshot of the entire page, obtain the image data of the web page, and preprocess the captured web page image data, the preprocessing includes denoising, geometric transformation, image enhancement and grayscale processing, the browser automation tool can be Selenium; the web page screenshot tool can be Pillow library, the captured web page image can be obtained by referring to Figure 3 As shown;

[0045] S2: construct a text recognition model, and train, test and verify the constructed text recognition model, and input the preprocessed web page image data into the verified text recognition model, and output the text content. The text recognition model can be a CRNN deep learning model for text recognition, and end-to-end text can be used;

[0046] S3: Construct a text line positioning model, and train, test and verify the constructed text line positioning model, input the output text content into the text line positioning model, and identify the position information of each line of text in the image, wherein the text line positioning model is used to realize text line positioning to determine the position information of each line of text in the image. Object detection models or text detection models, such as YOLO or EAST, can be used;

[0047] S4: Build a process function detection model, train, test and verify the built process function detection model, identify various functional elements in the web page based on the verified process function detection model, and build a deep learning model for process function detection to identify various functional elements in the web page, such as buttons, input boxes, links, etc. Use the target detection model, which can be Faster R-CNN or SSD;

[0048] S5: Based on the output results of the text recognition model and the process function detection model, write an automated test script to perform automated testing on the web page and generate an automated test report. Based on the output results of the text recognition model and the process function detection model, write an automated test script to achieve automated control of the web page. For example, click a button according to the identified button position, enter text according to the identified input box position, etc. After the test is completed, generate a test report to record the test results, error information, test coverage and other indicators for developers to analyze and optimize.

[0049] First, the web page interface is understood through image analysis, which is more flexible than the DOM analysis of Selenium in the prior art. Because it does not need to rely on the web page source code or DOM structure, it can handle various types and styles of web pages without being affected by changes in the web page structure. It can handle complex content in web page image data, such as charts, animations, etc., while Selenium in the prior art mainly targets the DOM structure of the web page and may not be able to effectively process these image contents. Using deep learning methods for image analysis, there is no need to access the source code of the web page, which means that it is possible to test dynamically generated web pages or when the web page source code is inaccessible, while Selenium needs to directly operate the web page DOM, which may not be possible in this case. Automatic control of web page elements through image analysis can more accurately locate and operate web page elements, while Selenium achieves automatic control by operating DOM elements, which may be affected by changes in the web page structure and lead to inaccurate positioning. The deep learning method is not limited by language and can process web page content in multiple languages, while Selenium may need to adapt to web pages in different languages.

[0050] In this embodiment, the specific process of preprocessing the intercepted web page image data in step S1 is as follows:

[0051] S11: performing denoising processing on the intercepted web page image data, deleting invalid image data and noise data therein, so as to prevent the invalid data or noise data therein from affecting the output results of subsequent models;

[0052] S12: geometrically transform the denoised web page image data to adjust the geometric properties of the web page image to consistency, including rotating, scaling, and translating the image data to adjust the geometric properties of the image to meet specific application requirements. These transformations can be achieved through coordinate transformation functions to ensure the geometric accuracy of the image;

[0053] S13: Use image enhancement processing such as contrast stretching, dynamic range compression, and grayscale slicing to highlight important image features in web images, improve image quality and interpretability. Image enhancement processing helps to improve image contrast and clarity, making images easier to analyze and understand.

[0054] After step S13, the image data is grayed out, and the color of each pixel in the web page image is represented by a gray value, the gray value range is 0-255, and the R, G, and B components in the color image are respectively used as the gray values ​​of the three gray images, or the maximum value of the three components is taken as the gray value, or the average value of the three components is calculated as the gray value.

[0055] Example 2

[0056] Based on Example 1, the specific process in step S2 is as follows:

[0057] S21: Obtain historical web page image data, annotate the historical web page image data, and construct a historical web page image data set, dividing the annotated web page image data into a training set, a test set, and a verification set;

[0058] S22: inputting the training set into the text recognition model to train it;

[0059] S23: input the test set into the text recognition model to test it, and output the test result, and determine whether the test result meets the preset test requirements. If so, execute step S24; if not, re-input a new training set for training;

[0060] S24: input the verification set into the text recognition model to verify its test results, and output the verification results to determine whether the verification results meet the preset verification requirements. If so, execute step S3; if not, re-input a new training set for training;

[0061] The process of training, testing and verifying the text line positioning model and the process function detection model in step S3 and step S4 is the same as the process of training, testing and verifying in the specific process of step S2.

[0062] Preferably, before training, testing and verifying the text recognition model, the text line positioning model and the process function detection model, the corresponding models are optimized, and the model optimization process is as follows:

[0063] Initialize the parameters of the text recognition model, text line location model, and process function detection model: including setting weights and biases, and initializing the first-order moment estimation variable m and the second-order moment estimation variable v, both of which have the same dimensions as the model parameters;

[0064] Compute sample gradients for text recognition models, text line localization models, and process function detection models;

[0065] The first-order moment estimation variable m and the second-order moment estimation variable v are updated: the first-order moment estimation of the current gradient, i.e., the average gradient, is calculated, m=β1m+(1-β1)g, and the second-order moment estimation variable v=β2v+(1-β2)g2, wherein β1 and β2 are adjustable exponential decay rates, and the values ​​are 0.9 and 0.999, respectively;

[0066] The first-order moment estimation variable m and the second-order moment estimation variable v are corrected: correcting the first-order moment estimation variable m, The second-order moment estimate variable v is corrected: Where t represents the current iteration number;

[0067] Update the model parameters of the text recognition model, text line location model, and process function detection model: estimate the variables based on the corrected first-order moment and the second moment estimation variable And the learning rate α updates the model parameters, θ represents the parameters of the model, ∈ is a very small number, take 10 -8 .

[0068] The functional elements in step S4 include titles, buttons, input boxes, links, forms, navigation bars, block elements, scripts, databases, programs, and footer blocks. The contents of the automated test report in step S5 include test environment, test objectives, test cases, execution status, defect list, and performance test results.

[0069] In summary, the web page automation testing method based on deep learning provided by the present invention uses a browser automation tool to access the web page to be tested, obtains the image data of the web page, and preprocesses the intercepted web page image data; constructs a text recognition model, and inputs the preprocessed web page image data into the verified text recognition model, and outputs the text content; constructs a text line positioning model, inputs the output text content into the text line positioning model, and identifies the position information of each line of text in the image; constructs a process function detection model, and identifies each functional element in the web page based on the verified process function detection model; writes an automated test script according to the output results of the text recognition model and the process function detection model, performs automated testing on the web page, and generates an automated test report.

[0070] First, the web page interface is understood through image analysis, which is more flexible than the DOM analysis of Selenium in the prior art. Because it does not need to rely on the web page source code or DOM structure, it can handle various types and styles of web pages without being affected by changes in the web page structure. It can handle complex content in web page image data, such as charts, animations, etc., while Selenium in the prior art mainly targets the DOM structure of the web page and may not be able to effectively process these image contents. Using deep learning methods for image analysis, there is no need to access the source code of the web page, which means that it is possible to test dynamically generated web pages or when the web page source code is inaccessible, while Selenium needs to directly operate the web page DOM, which may not be possible in this case. Automatic control of web page elements through image analysis can more accurately locate and operate web page elements, while Selenium achieves automatic control by operating DOM elements, which may be affected by changes in the web page structure and lead to inaccurate positioning. The deep learning method is not limited by language and can process web page content in multiple languages, while Selenium may need to adapt to web pages in different languages.

Claims

1. A web page automated testing method based on deep learning, characterized in that: The following steps are involved: S1: Use a browser automation tool to access the web page to be tested, and use a screenshot tool to take a screenshot of the entire page, obtain image data of the web page, and pre-process the captured web page image data; S2: construct a text recognition model, and train, test and verify the constructed text recognition model, input the preprocessed web page image data into the verified text recognition model, and output text content; S3: constructing a text line positioning model, and training, testing and verifying the constructed text line positioning model, inputting the output text content into the text line positioning model, and identifying the position information of each line of text in the image; S4: construct a process function detection model, and train, test and verify the constructed process function detection model, and identify various functional elements in the web page based on the verified process function detection model; S5: Based on the output results of the text recognition model and the process function detection model, write automated testing scripts, perform automated testing on web pages, and generate automated testing reports.

2. According to the method of automatic web page testing based on deep learning in claim 1, it is characterized in that: The specific process of preprocessing the intercepted web page image data in step S1 is as follows: S11: performing denoising processing on the intercepted web page image data to delete invalid image data and noise data therein; S12: performing geometric transformation processing on the web page image data after the denoising process to adjust the geometric properties of the web page image to consistency, including rotating, scaling, and translating the image data; S13: Use contrast stretching, dynamic range compression, and grayscale slicing to enhance the image and highlight important image features in web images.

3. The webpage automated testing method based on deep learning according to claim 2, characterized in that: After step S13, the image data is grayed out, and the color of each pixel in the web page image is represented by a gray value, the gray value range is 0-255, and the R, G, and B components in the color image are respectively used as the gray values ​​of the three gray images, or the maximum value of the three components is taken as the gray value, or the average value of the three components is calculated as the gray value.

4. The webpage automated testing method based on deep learning according to claim 1, characterized in that: The specific process in step S2 is as follows: S21: Obtain historical web page image data, annotate the historical web page image data, and construct a historical web page image data set, dividing the annotated web page image data into a training set, a test set, and a verification set; S22: inputting the training set into the text recognition model to train it; S23: input the test set into the text recognition model to test it, and output the test result, and determine whether the test result meets the preset test requirements. If so, execute step S24; if not, re-input a new training set for training; S24: input the verification set into the text recognition model to verify its test results, and output the verification results to determine whether the verification results meet the preset verification requirements. If so, execute step S3; if not, re-input a new training set for training; The process of training, testing and verifying the text line positioning model and the process function detection model in step S3 and step S4 is the same as the process of training, testing and verifying in the specific process of step S2.

5. The webpage automated testing method based on deep learning according to claim 4, characterized in that: Before training, testing, and verifying the text recognition model, text line location model, and process function detection model, the corresponding models are optimized. The model optimization process is as follows: Initialize the parameters of the text recognition model, text line location model, and process function detection model: including setting weights and biases, and initializing the first-order moment estimation variable m and the second-order moment estimation variable v, both of which have the same dimensions as the model parameters; Compute sample gradients for text recognition models, text line localization models, and process function detection models; The first-order moment estimation variable m and the second-order moment estimation variable v are updated: the first-order moment estimation of the current gradient, i.e., the average gradient, is calculated, m=β1m+(1-β1)g, and the second-order moment estimation variable v=β2v+(1-β2)g2, wherein β1 and β2 are adjustable exponential decay rates, and the values ​​are 0.9 and 0.999, respectively; The first-order moment estimation variable m and the second-order moment estimation variable v are corrected: correcting the first-order moment estimation variable m, The second-order moment estimate variable v is corrected: Where t represents the current iteration number; Update the model parameters of the text recognition model, text line location model, and process function detection model: estimate the variables based on the corrected first-order moment and the second moment estimation variable And the learning rate α updates the model parameters, θ represents the parameters of the model, ∈ is a very small number, take 10 -8 .

6. The webpage automated testing method based on deep learning according to claim 1, characterized in that: The functional elements in step S4 include titles, buttons, input boxes, links, forms, navigation bars, block elements, scripts, databases, programs, and footer blocks.

7. The webpage automated testing method based on deep learning according to claim 1, characterized in that: The content of the automated test report in step S5 includes the test environment, test objectives, test cases, execution status, defect list, and performance test results.

Images