Method and system for generating and auditing automatic test result of set top box

Abstract

The invention discloses a method and a system for generating and auditing an automatic test result of a set top box, and mainly relates to the technical field of automatic test of the set top box. Comprising the steps of automatically generating assertion judgment and executing assertion judgment, and the step of automatically generating assertion judgment comprises the following sub-steps: performing screenshot on a current page of equipment by using an ADB debugging bridge, and transmitting a picture to a test platform; storing the picture as a picture element, and marking the picture element by using a test case name and a number as a judgment basis during script execution; and a step of adding an assertion judgment to the current script, taking the picture elements as a comparison basis, and setting a default similarity matching threshold. The method has the advantages that script execution and result judgment can be well combined, which step has problems can be traced through a test report, and specific reasons can be conveniently checked.

Description

Technical Field

[0001] The present invention relates to the technical field of set-top box automated testing, and in particular to a method and system for generating and reviewing automated testing results of a set-top box. Background Art

[0002] Due to the particularity of set-top box devices and the limitations of usage scenarios, testing them mainly relies on manual work; the existing automated testing frameworks are already very mature in mobile devices and web pages, but they are not compatible with set-top box testing. Even if some can be barely used, it is impossible to test multiple devices at the same time. There are three main reasons for this problem. First, due to the characteristics of the set-top box device, the operation page is operated through a remote control, and it cannot be touched, clicked, or slid like a mobile phone; second, the configuration and performance of most IPTV set-top boxes are relatively general, and cannot support large-scale automated testing; third, due to the low configuration and system version of most TV set-top boxes, most of them are based on the Android 4.4 system, and the customized development of various manufacturers, the framework is difficult to be compatible. In this way, the page content cannot be operated and judged through mainstream methods such as control ID or Xpath. Even if the script can be executed with commands through methods such as ADB debug bridge, there is no particularly good solution to judge the results of script execution. It is necessary to manually observe the HDMI output to determine whether the picture is normal. If it fails, it can only be re-executed.

[0003] At present, when performing automated testing on set-top box devices, it is impossible to use a mature framework for assertion judgment. Even if the devices can be controlled in batches, it can only ensure that the devices can run. As for whether the operation is correct or not, and which step of the operation has a problem, it is impossible to trace it. This leads to a poor combination of script execution and result judgment. In addition, the device may have misjudgments: 1. Mistakenly judging wrong results as passed. When the judgment conditions are not strict enough and the minimum similarity threshold is set low, wrong results may be overlooked. 2. Because the conditions are too strict, two pictures with the same content may be judged as failed due to factors such as different execution devices, network loading, and changes in the page carousel.

[0004] Therefore, there is an urgent need for a method and system for generating and reviewing automatic test results of a set-top box to solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for generating and reviewing set-top box automated test results, which can well combine script execution and result determination, and can also trace the step where the problem occurs through the test report, making it convenient to troubleshoot the specific cause.

[0006] In order to achieve the above object, the present invention is implemented through the following technical solutions: On the one hand, the present invention provides a method for generating and reviewing automatic test results of a set-top box, including automatically generating an assertion judgment and executing the assertion judgment, wherein the automatically generating the assertion judgment comprises the following steps: Use ADB debug bridge to take a screenshot of the current page of the device and transfer the picture to the test platform; Store the image as an image element and mark it with the test case name and number as the basis for judgment when the script is executed; Add an assertion step to the current script, use the above image elements as the basis for comparison, and set the default similarity matching threshold.

[0007] Preferably, the execution assertion judgment comprises the following steps: S1: Use ADB debug bridge to take a screenshot of the current page of the device and transfer the picture to the test platform; S2: Get the image elements saved during script recording and the real-time image captured by the current assertion step; S3: Calculate the similarity of the two images using the structural similarity algorithm and the histogram similarity algorithm respectively.

[0008] Preferably, in step S3, the structural similarity algorithm includes: Check whether the two image files in1 and in2 passed in are empty. If any of the files is empty, the function returns 0, indicating that the calculation cannot be performed; Use the OpenCV function Imgcodecs.imread to read the two image files in1 and in2 passed in and convert them into grayscale image matrix data mat1 and mat2; Use OpenCV's Imgproc.resize function to resize the two images so that they are the same size; The match between the two grayscale images mat1 and mat2 is calculated by the Imgproc.matchTemplate function of OpenCV, which uses a template matching algorithm and saves the result in ssimMat; Use OpenCV's Core.mean function to calculate the mean in ssimMat, which represents the similarity between the two images. Core.mean(ssimMat).val[0] returns the mean of ssimMat. If the value is less than 0, it means that the image match is low, and 0 is returned; otherwise, the calculated similarity value is returned.

[0009] Preferably, in step S3, the histogram similarity algorithm includes: Check whether the two input image files in1 and in2 are empty. If any file is empty, it returns 0, indicating that subsequent calculations cannot be performed; Use the Imgcodecs.imread method provided by OpenCV to read the image files in1 and in2 and convert them into matrix data mat1 and mat2; Use OpenCV's Imgproc.resize function to resize the two images so that they are the same size; Use OpenCV's Imgproc.cvtColor method to convert the RGB color space to the HSV color space; Calculate the histograms of the two images separately: By calling OpenCV's Imgproc.calcHist, select hue as the analysis dimension, and store the calculation results in the corresponding matrix. The histogram calculation range is [0,255], which is divided into 255 histogram intervals.

[0010] Use OpenCV's Core.normalize method to normalize the histogram: the normalization uses minimum-maximum normalization to normalize the histogram data range to [1, rows]; Use OpenCV's Imgproc.compareHist method to compare the histograms of the two images. Use correlation as the comparison method. The correlation value range is [-1,1], where 1 indicates complete correlation, 0 indicates no correlation, and -1 indicates complete negative correlation. Determine whether the histogram similarity score is less than 0: If the score is less than 0, return 0, indicating that the two images have no similarity; if the score is greater than or equal to 0, return the calculated similarity score.

[0011] Preferably, the execution assertion judgment further includes the following steps: S4: Take the smaller value of the results of the structural similarity algorithm and the histogram similarity algorithm; S5: The final calculation result of the algorithm is compared with the minimum similarity threshold set in the assertion step. If the result is higher than the threshold, the system determines that the two images are the same, and this step test passes; if the result is lower than the threshold, the system determines that the two images are different, and this step test fails; S6: After the script is executed, the system will automatically review and classify the abnormalities and error screenshots that occurred during the test into a picture library. The system will match the screenshots that failed the test with the error screenshots in the picture library. If the match is successful, it means that there is no misjudgment and this step has failed. If the match fails, it is considered that this step has been misjudged and will be corrected to pass the test; S7: After the review is completed, a test report is generated, in which the judgment results of each assertion affect the final result.

[0012] On the other hand, a system based on the above-mentioned method for generating and reviewing the automated test results of a set-top box is provided, comprising: The data acquisition module is used to: use the ADB debugging bridge to take a screenshot of the current page of the device and transfer the picture to the test platform; The data processing module is used to: store the image as an image element and mark it with the test case name and number as the judgment basis when the script is executed; The data output module is used to: add an assertion judgment step in the current script, use the image elements in step S2 as the basis for comparison, and set a default similarity matching threshold.

[0013] Compared with the prior art, the beneficial effects of the present invention are: 1. The present invention is based on OpenCV. When recording the script before the test, it can take screenshots, store image elements, add assertion steps and other operations for important nodes that need to be judged with one click; when the automated script executes to the assertion step, it takes a screenshot of the device and compares it with the set image elements, and uses the structural similarity algorithm and the histogram similarity algorithm to calculate the similarity respectively. The smaller value of the two calculation results is taken, and then judged with the minimum similarity threshold set by the assertion to obtain a preliminary judgment conclusion. The judgment conclusion of all assertion nodes is a complete test report, which can well combine script execution and result judgment, and can also trace the problem at which step through the test report, so as to facilitate the investigation of specific causes.

[0014] 2. The present invention combines the two algorithms and takes the smaller value and sets the threshold to a higher level, which restricts the machine judgment and almost eliminates the problem of errors being missed.

[0015] 3. After the script is executed, the present invention will automatically review it. The system has taken screenshots and classified the screens where problems occurred during the previous manual test. During the review, the screenshots that failed the test will be compared with the error screenshot library. If they match, it means that it is indeed an error. If they do not match, it is judged as a misjudgment and the result is automatically corrected. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is the automatic generation of assertion judgment flow chart of the present invention; Figure 2 is the execution assertion judgment flow chart of the present invention; Figure 3 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0017] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the application equally.

[0018] In the present invention, terms such as "upper", "lower", "left", "right", "front", "back", "vertical", "horizontal", "side", "bottom", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings. They are relational words determined only for the convenience of describing the structural relationships of the various parts or elements of the present invention, and do not specifically refer to any part or element in the present invention and should not be understood as limitations on the present invention.

[0019] Example: like Figure 1 As shown, this embodiment provides a method for generating and reviewing automated test results for a set-top box, by setting assertions at key nodes of script execution to determine whether a test point has passed. The assertion setting is generated with one click during the script recording phase. It is common to connect the device through the ADB debug bridge. Based on this premise, the specific steps will not be described, and the subsequent steps will be divided into assertion setting and assertion judgment for detailed description.

[0020] When recording a script, press the Assertion button to automatically generate an assertion judgment according to the following steps: Use ADB debug bridge to take a screenshot of the current page of the device and transfer the picture to the test platform; Store the image as an image element and mark it with the test case name and number as the basis for judgment when the script is executed; Add an assertion step to the current script, use the above image elements as the basis for comparison, and set the default similarity matching threshold.

[0021] The above is the breakdown of the process of setting assertions. For users, it is generated with one click. They only need to add this assertion in all places where judgment is required.

[0022] The process of assertion judgment is more complicated, such as Figure 2 As shown in the figure, when the test script executes to the assertion step, the following operations are performed: S1: Use ADB debug bridge to take a screenshot of the current page of the device and transfer the picture to the test platform; S2: Get the image elements saved during script recording and the real-time image captured by the current assertion step; S3: Calculate the similarity of the two images using the structural similarity algorithm and the histogram similarity algorithm respectively; The structural similarity algorithm is divided into the following steps: (1) First, check whether the two image files (in1 and in2) passed in are empty. If either file is empty, the function returns 0, indicating that the calculation cannot be performed; (2) Use the OpenCV function Imgcodecs.imread to read the two image files in1 and in2 and convert them into grayscale image matrix data mat1 and mat2. The purpose of converting them into grayscale images is to simplify the calculation of SSIM (Structural Similarity Index, an indicator used to evaluate the visual similarity of two images), because SSIM is usually calculated on grayscale images and ignores color information; (3) Use OpenCV's Imgproc.resize function to resize the two images so that they are the same size so that similarity comparison can be performed; (4) Calculate the matching degree between the two grayscale images (mat1 and mat2) through the Imgproc.matchTemplate function of OpenCV. This function uses the template matching algorithm and saves the result in ssimMat. The matching method here uses Imgproc.CC_STAT_HEIGHT; (5) Use OpenCV's Core.mean function to calculate the mean in ssimMat, which represents the similarity between the two images. Core.mean(ssimMat).val[0] returns the mean of ssimMat. If the value is less than 0, it means that the image matching degree is low, and 0 is returned; otherwise, the calculated similarity value is returned. The specific code is as follows: publicstaticdoublecalculateSSIM(Filein1,Filein2){ if(null==in1||null==in2){ return0; } Matmat1=Imgcodecs.imread(in1.getAbsolutePath(),Imgcodecs.IMREAD_GRAYSCALE); Matmat2=Imgcodecs.imread(in2.getAbsolutePath(),Imgcodecs.IMREAD_GRAYSCALE); Imgproc.resize(mat1,mat1,mat2.size()); Imgproc.resize(mat2,mat2,mat1.size()); MatOfFloatssimMat=newMatOfFloat(); Imgproc.matchTemplate(mat1,mat2,ssimMat,Imgproc.CC_STAT_HEIGHT); doublev=Core.mean(ssimMat).val[0]; if(v<0){ return0; } returnv; } The histogram similarity algorithm consists of the following steps: (1) First, check whether the two input image files (in1 and in2) are empty. If either file is empty, 0 is returned, indicating that subsequent calculations cannot be performed; (2) Use the Imgcodecs.imread method provided by OpenCV to read the image files in1 and in2 and convert them into matrix data mat1 and mat2. The images are in the original data format that retains the RGB color space, providing a basis for subsequent processing; (3) Use OpenCV's Imgproc.resize function to resize the two images so that they are the same size so that similarity comparison can be performed; (4) Use the Imgproc.cvtColor method of OpenCV to convert the RGB color space to the HSV color space. The HSV color space divides colors into hue, saturation, and value. Hue can best reflect the main color information of the image. By converting to the HSV space, the color distribution characteristics of the image can be captured more efficiently and the sensitivity to brightness changes can be reduced. (5) Calculate the histograms of the two images separately: By calling OpenCV's Imgproc.calcHist, select hue (H channel) as the analysis dimension and store the calculation results in the corresponding matrix. The histogram calculation range is [0, 255], which is divided into 255 histogram intervals; (6) Use OpenCV's Core.normalize method to normalize the histogram. The normalization adopts the minimum-maximum normalization (Core.NORM_MINMAX) to normalize the data range of the histogram to [1, rows]. The purpose of normalization is to eliminate the influence of image brightness, number of pixels or resolution differences on the distribution of histogram data, thereby improving the robustness of the comparison; (7) Use OpenCV's Imgproc.compareHist method to compare the histograms of the two images, using correlation as the comparison method (Imgproc.CV_COMP_CORREL). The correlation value range is [-1, 1], where 1 indicates complete correlation, 0 indicates no correlation, and -1 indicates complete negative correlation; (8) Determine whether the histogram similarity score is less than 0: If the score is less than 0, return 0, indicating that the two images have no similarity; if the score is greater than or equal to 0, return the calculated similarity score. A high score indicates that the color distribution of the two images is more similar. The specific code is as follows: publicstaticdoublecalculateHistogram(Filein1,Filein2){ if(null==in1||null==in2){ return0; } Matmat1=Imgcodecs.imread(in1.getAbsolutePath()); Matmat2=Imgcodecs.imread(in2.getAbsolutePath()); / / High resolution processing as low resolution Imgproc.resize(mat1,mat1,mat2.size()); Imgproc.resize(mat2,mat2,mat1.size()); / / Color space conversion Imgproc.cvtColor(mat1,mat1,Imgproc.COLOR_RGB2HSV); Imgproc.cvtColor(mat2,mat2,Imgproc.COLOR_RGB2HSV); / / Calculate the histogram Imgproc.calcHist(List.of(mat1),newMatOfInt(0),newMat(),mat1,newMatOfInt(255),newMatOfFloat(0,255)); Imgproc.calcHist(List.of(mat2),newMatOfInt(0),newMat(),mat2,newMatOfInt(255),newMatOfFloat(0,255)); / / Normalization Core.normalize(mat1,mat1,1,mat1.rows(),Core.NORM_MINMAX,-1,newMat()); Core.normalize(mat2,mat2,1,mat2.rows(),Core.NORM_MINMAX,-1,newMat()); doublescore=Imgproc.compareHist(mat1,mat2,Imgproc.CV_COMP_CORREL); if(score<0){ return0; } returnscore; } S4: Take the smaller value of the results of the structural similarity algorithm and the histogram similarity algorithm. This can reduce the judgment error and avoid the situation where the images with small differences are missed; S5: The final calculation result of the algorithm is compared with the minimum similarity threshold set in the assertion step. If the result is higher than the threshold, the system determines that the two images are the same, and this step test passes; if the result is lower than the threshold, the system determines that the two images are different, and this step test fails; S6: After the script is executed, the system will automatically review it. Here, you need to collect and classify various abnormalities and error screenshots that occurred during the previous manual test into a picture library in advance. The system will match the screenshots that failed this test with the error screenshots in the picture library. If the match is successful, it means that there is no misjudgment and this step has failed. If the match fails, it is considered that this step has been misjudged and will be corrected to pass the test; S7: After the review is completed, a test report will be generated. The final result is affected by the judgment results of each assertion. The report will show the original image elements, device screenshots during execution, similarity determined by the system, and system judgment results (pass / fail). The final test conclusion will be generated based on whether the assertion node has a failed status. When all the assertion nodes in the system judgment + review pass, the script is executed successfully and the device passes the test.

[0023] Although the above image comparison method and algorithm are relatively demanding, they can prevent most systems from misjudging incorrect results as passed. In the actual test process, when the minimum similarity threshold is set to about 80%, the test conclusion is at a high accuracy and the workload of review is relatively small. It is a relatively balanced value, but it cannot be copied. It needs to be tested multiple times to confirm the reasonable threshold range.

[0024] like Figure 3 As shown, this embodiment also provides a system based on the above-mentioned method for generating and reviewing the automatic test results of the set-top box, including: The data acquisition module is used to: use the ADB debugging bridge to take a screenshot of the current page of the device and transfer the picture to the test platform; The data processing module is used to: store the image as an image element and mark it with the test case name and number as the judgment basis when the script is executed; The data output module is used to: add an assertion judgment step in the current script, use the image elements in step S2 as the basis for comparison, and set a default similarity matching threshold.

[0025] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for generating and reviewing automated test results of a set-top box, comprising automatically generating assertion judgments and executing assertion judgments, characterized in that: The automatic generation of assertion judgment comprises the following steps: Use ADB debug bridge to take a screenshot of the current page of the device and transfer the picture to the test platform; Store the image as an image element and mark it with the test case name and number as the basis for judgment when the script is executed; Add an assertion step to the current script, use the above image elements as the basis for comparison, and set the default similarity matching threshold.

2. The method for generating and reviewing set-top box automated test results according to claim 1, characterized in that: The execution assertion judgment comprises the following steps: S1: Use ADB debug bridge to take a screenshot of the current page of the device and transfer the picture to the test platform; S2: Get the image elements saved during script recording and the real-time image captured by the current assertion step; S3: Calculate the similarity of the two images using the structural similarity algorithm and the histogram similarity algorithm respectively.

3. The method for generating and reviewing the automatic test results of a set-top box according to claim 2, characterized in that: In step S3, the structural similarity algorithm includes: Check whether the two image files in1 and in2 passed in are empty. If any of the files is empty, the function returns 0, indicating that the calculation cannot be performed; Use the OpenCV function Imgcodecs.imread to read the two image files in1 and in2 passed in and convert them into grayscale image matrix data mat1 and mat2; Use OpenCV's Imgproc.resize function to resize the two images so that they are the same size; The match between the two grayscale images mat1 and mat2 is calculated by the Imgproc.matchTemplate function of OpenCV, which uses a template matching algorithm and saves the result in ssimMat; Use OpenCV's Core.mean function to calculate the mean in ssimMat, which represents the similarity between the two images. Core.mean(ssimMat).val[0] returns the mean of ssimMat. If the value is less than 0, it means that the image match is low, and 0 is returned; otherwise, the calculated similarity value is returned.

4. The method for generating and reviewing the automatic test results of a set-top box according to claim 2, characterized in that: In step S3, the histogram similarity algorithm includes: Check whether the two input image files in1 and in2 are empty. If any file is empty, it returns 0, indicating that subsequent calculations cannot be performed; Use the Imgcodecs.imread method provided by OpenCV to read the image files in1 and in2 and convert them into matrix data mat1 and mat2; Use OpenCV's Imgproc.resize function to resize the two images so that they are the same size; Use OpenCV's Imgproc.cvtColor method to convert the RGB color space to the HSV color space; Calculate the histograms of the two images separately: by calling OpenCV's Imgproc.calcHist, select hue as the analysis dimension, and store the calculation results in the corresponding matrix. The histogram calculation range is [0,255], which is divided into 255 histogram intervals; Use OpenCV's Core.normalize method to normalize the histogram: the normalization uses minimum-maximum normalization to normalize the histogram data range to [1, rows]; Use OpenCV's Imgproc.compareHist method to compare the histograms of the two images. Use correlation as the comparison method. The correlation value range is [-1,1], where 1 indicates complete correlation, 0 indicates no correlation, and -1 indicates complete negative correlation. Determine whether the histogram similarity score is less than 0: If the score is less than 0, return 0, indicating that the two images have no similarity; if the score is greater than or equal to 0, return the calculated similarity score.

5. The method for generating and reviewing set-top box automated test results according to claim 2, characterized in that: The execution assertion judgment also includes the following steps: S4: Take the smaller value of the results of the structural similarity algorithm and the histogram similarity algorithm; S5: The final calculation result of the algorithm is compared with the minimum similarity threshold set in the assertion step. If the result is higher than the threshold, the system determines that the two images are the same, and this step test passes; if the result is lower than the threshold, the system determines that the two images are different, and this step test fails; S6: After the script is executed, the system will automatically review and classify the abnormalities and error screenshots that occurred during the test into a picture library. The system will match the screenshots that failed the test with the error screenshots in the picture library. If the match is successful, it means that there is no misjudgment and this step has failed. If the match fails, it is considered that this step has been misjudged and will be corrected to pass the test; S7: After the review is completed, a test report is generated, in which the judgment results of each assertion affect the final result.

6. A system based on the method for generating and reviewing the set-top box automated test results as claimed in claim 1, characterized in that: include: The data acquisition module is used to: use the ADB debugging bridge to take a screenshot of the current page of the device and transfer the picture to the test platform; The data processing module is used to: store the image as an image element and mark it with the test case name and number as the judgment basis when the script is executed; The data output module is used to: add an assertion judgment step in the current script, use the image elements in step S2 as the basis for comparison, and set a default similarity matching threshold.

Images