User interface test method and device, computer equipment and storage medium
By calculating the similarity score between the user interface test images and the standard design images, identifying the differences area, the problems of low efficiency and high cost of traditional user interface testing methods are solved, and efficient and accurate user interface testing is achieved.
Patent Information
- Application Number
- CN202411014179.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-12
AI Technical Summary
Traditional user interface testing methods are inefficient and costly, making it difficult to effectively evaluate the similarity between user interface images and standard design images.
By acquiring user interface test images and standard design images, the structure similarity index and pixel value variance are calculated after image channel fusion processing, and a comprehensive similarity score is generated to characterize the image similarity level, and image fusion and labeling are performed according to the score. Identify the difference area.
It improves the efficiency and accuracy of user interface testing, reduces the testing cost, and balances different image characteristics through similarity evaluation, which enhances the visibility and convenience of the test.
Smart Images

Figure CN118939549A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of software technology, and in particular to a user interface testing method, apparatus, computer equipment and storage medium. Background Art
[0002] The Graphic User Interface (GUI) is the main way for computer software to interact with users. In order to ensure that the developed GUI can run normally, it needs to be tested during the GUI development process.
[0003] However, in traditional user interface testing methods, the elements in the GUI page are generally tested separately through test cases, which has problems such as low testing efficiency or high testing cost. Summary of the invention
[0004] Based on this, it is necessary to provide a user interface testing method, apparatus, computer equipment and storage medium that can improve testing efficiency and reduce testing costs in response to the above technical problems.
[0005] In a first aspect, a user interface testing method is provided, the method comprising:
[0006] Obtaining a user interface test image and a user interface standard design image of an application;
[0007] Performing image channel fusion processing on the user interface test image and the user interface standard design image respectively to obtain a user interface test grayscale image and a user interface standard design grayscale image;
[0008] Determine a first image structure similarity index and a first image pixel value variance according to a user interface test grayscale image, and determine a second image structure similarity index and a second image pixel value variance according to a user interface standard design grayscale image;
[0009] A calculation is performed based on the first image structure similarity index, the second image structure similarity index, the first image pixel value variance and the second image pixel value variance to obtain a similarity comprehensive score of the user interface test image; wherein the similarity comprehensive score is used to characterize the degree of similarity between the user interface test image and the user interface standard design image.
[0010] In one of the embodiments, the method includes: in response to a comprehensive similarity score being greater than a score threshold, performing image fusion processing on a user interface test image and a user interface standard design image to obtain a user interface fusion image; performing image processing on the user interface fusion image to obtain target difference regions; wherein the target difference regions are regions where differences exist between the user interface test image and the user interface standard design image; and annotating the user interface fusion image according to the target difference regions to obtain an annotated user interface fusion image.
[0011] In one of the embodiments, image processing is performed on the user interface fusion image to obtain the target difference area, including: multiplying the pixel value of each pixel point of the user interface fusion image by the maximum pixel value to obtain the pixel value of each processed pixel point; converting the pixel value of each processed pixel point into an unsigned eight-bit integer data format to obtain the pixel value of each pixel point in a standard format; based on the maximum between-class variance method, performing image binarization processing on the user interface fusion image according to the pixel values of each pixel point in the standard format to obtain the user interface fusion image after binarization processing; and determining each target difference area according to the black area in the user interface fusion image after binarization processing.
[0012] In one of the embodiments, the user interface fusion image is annotated according to the target difference areas to obtain the annotated user interface fusion image, including: determining the outline of the corresponding target difference area according to each target difference area; calculating according to the outline of each target difference area to obtain the minimum bounding rectangle of the corresponding target difference area; annotating the user interface fusion image according to each minimum bounding rectangle to obtain the annotated user interface fusion image.
[0013] In one embodiment, the method includes: generating a portable network graphics format file according to the annotated user interface fusion image.
[0014] In one of the embodiments, obtaining a user interface test image of an application includes: determining a user interface uniform resource locator system address in response to an input operation on the user interface uniform resource locator system address of the application; and taking a screenshot of a web page interface according to the user interface uniform resource locator system address based on an automated testing tool to obtain a user interface test image.
[0015] In one embodiment, a first image structure similarity index and a first image pixel value variance are determined based on a user interface test grayscale image, and a second image structure similarity index and a second image pixel value variance are determined based on a user interface standard design grayscale image, including: an image structure similarity index calculation function based on an image processing database, calculating the first image structure similarity index based on the user interface test grayscale image, and calculating the second image structure similarity index based on the user interface standard design grayscale image.
[0016] In one embodiment, a first image structure similarity index and a first image pixel value variance are determined based on a user interface test grayscale image, and a second image structure similarity index and a second image pixel value variance are determined based on a user interface standard design grayscale image, including: calculating the first image pixel value variance based on the total number of pixels in the user interface test grayscale image, the brightness value of each pixel, and the average brightness value of all pixels; calculating the second image pixel value variance based on the total number of pixels in the user interface standard design grayscale image, the brightness value of each pixel, and the average brightness value of all pixels.
[0017] In one of the embodiments, a calculation is performed based on the first image structure similarity index, the second image structure similarity index, the first image pixel value variance and the second image pixel value variance to obtain a comprehensive similarity score of the user interface test image, including: determining a similarity index ratio based on the quotient of the first image structure similarity index and the second image structure similarity index; determining a pixel value variance ratio based on the quotient of the first image pixel value variance and the second image pixel value variance; and performing individual calculations based on the similarity index ratio, the weight of the similarity index ratio, the pixel value variance ratio and the weight of the pixel value variance ratio to obtain a comprehensive similarity score.
[0018] In a second aspect, a user interface testing device is provided, which includes an image acquisition module, an image channel fusion module, a parameter calculation module and a score calculation module.
[0019] Among them, the image acquisition module is used to acquire the user interface test image and the user interface standard design image of the application; the image channel fusion module is used to perform image channel fusion processing on the user interface test image and the user interface standard design image respectively, so as to obtain the user interface test grayscale image and the user interface standard design grayscale image; the parameter calculation module is used to determine the first image structure similarity index and the first image pixel value variance according to the user interface test grayscale image, and determine the second image structure similarity index and the second image pixel value variance according to the user interface standard design grayscale image; the score calculation module is used to calculate according to the first image structure similarity index, the second image structure similarity index, the first image pixel value variance and the second image pixel value variance, so as to obtain the comprehensive similarity score of the user interface test image; wherein, the comprehensive similarity score is used to characterize the degree of similarity between the user interface test image and the user interface standard design image.
[0020] In a third aspect, a computer device is provided. The computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps of any method in the above method embodiments are implemented.
[0021] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any method in the above method embodiments are implemented.
[0022] The user interface test method, apparatus, computer equipment and storage medium described above obtain a user interface test image and a user interface standard design image of an application; then, image channel fusion processing is performed on the user interface test image and the user interface standard design image respectively to obtain a user interface test grayscale image and a user interface standard design grayscale image; then, a first image structure similarity index and a first image pixel value variance are determined according to the user interface test grayscale image, and a second image structure similarity index and a second image pixel value variance are determined according to the user interface standard design grayscale image; finally, a similarity comprehensive score of the user interface test image is calculated according to the first image structure similarity index, the second image structure similarity index, the first image pixel value variance and the second image pixel value variance; wherein the similarity comprehensive score is used to characterize the similarity between the user interface test image and the user interface standard design image, so that the image similarity can be comprehensively evaluated through the similarity comprehensive score, the importance of different image characteristics can be balanced, the test efficiency and test accuracy can be improved, and the test cost can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 An application environment diagram of a user interface testing method in an embodiment;
[0024] Figure 2 A first flow chart of a user interface testing method in one embodiment;
[0025] Figure 3 A schematic diagram of a process for obtaining a user interface test image of an application in one embodiment;
[0026] Figure 4 A schematic diagram of a flow chart of obtaining a similarity comprehensive score of a user interface test image by calculating according to a first image structure similarity index, a second image structure similarity index, a first image pixel value variance, and a second image pixel value variance in an embodiment;
[0027] Figure 5 A second flow chart of a user interface testing method in one embodiment;
[0028] Figure 6 A schematic diagram of a process of performing image processing on a user interface fusion image to obtain a target difference area in an embodiment;
[0029] Figure 7 It is a schematic diagram of a process of annotating a user interface fusion image according to a target difference area to obtain an annotated user interface fusion image in one embodiment;
[0030] Figure 8 is a structural block diagram of a user interface testing device in one embodiment;
[0031] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0032] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0033] In order to facilitate understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. Embodiments of the present application are provided in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.
[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.
[0035] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first resistor may be referred to as a second resistor, and similarly, a second resistor may be referred to as a first resistor. Both the first resistor and the second resistor are resistors, but they are not the same resistor.
[0036] It can be understood that the “connection” in the following embodiments should be understood as “electrical connection”, “communication connection”, etc. if the connected circuits, modules, units, etc. have electrical signals or data transmission between each other.
[0037] When used herein, the singular forms "a", "an", and "said / the" may also include plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "include / comprise" or "have" etc. specify the presence of stated features, wholes, steps, operations, components, parts or combinations thereof, but do not exclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts or combinations thereof.
[0038] The user interface testing method provided in this application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices, and the server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.
[0039] First, as Figure 2 As shown, a user interface testing method is provided, which is applied to Figure 1 The server 104 in the example is used for explanation, and the steps include the following steps 201 to 204.
[0040] Step 201: Acquire a user interface test image and a user interface standard design image of an application.
[0041] The application is the application to be tested. The user interface test image is the image to be tested of the entire user interface of the application. The user interface standard design image is the design draft image of the entire user interface of the application. Specifically, the server 104 can obtain the user interface test image and the user interface standard design image of the application.
[0042] In one embodiment, if Figure 3 As shown, obtaining a user interface test image of an application includes steps 301 to 302.
[0043] Step 301 : In response to an input operation on a user interface uniform resource locator system address of an application, determine a user interface uniform resource locator system address.
[0044] The user interface uniform resource locator (URL) is the user interface uniform resource locator (URL). When the server 104 recognizes the input operation on the user interface uniform resource locator (URL) of the application, it can determine the user interface uniform resource locator (URL) according to the result of the input operation.
[0045] Step 302: Based on the automated testing tool, a web page interface screenshot is processed according to the user interface uniform resource locator system address to obtain a user interface test image.
[0046] Specifically, the server 104 can process the web page interface screenshot based on the user interface uniform resource locator system address based on the automated testing tool to obtain the user interface test image, thereby improving the efficiency and convenience of obtaining the user interface test image.
[0047] In a specific example, the automated testing tool may be, but is not limited to, the page.screenshot program of the Playwrite database. The above is only a specific example, and in actual applications, it can be flexibly set according to user needs and is not limited here.
[0048] In this embodiment, in response to an input operation on the user interface uniform resource locator system address of the application, the user interface uniform resource locator system address is determined; then, based on an automated testing tool, a web page interface screenshot is processed according to the user interface uniform resource locator system address to obtain a user interface test image, thereby improving the efficiency and convenience of obtaining the user interface test image and ensuring the integrity and consistency of the test data.
[0049] In a specific example, obtaining a user interface test image and a user interface standard design image of an application includes: in response to an upload operation on the user interface standard design image, obtaining the user interface standard design image. The above is only a specific example, and in actual applications it can be flexibly set according to user needs and is not limited here.
[0050] Step 202 , performing image channel fusion processing on the user interface test image and the user interface standard design image respectively to obtain a user interface test grayscale image and a user interface standard design grayscale image.
[0051] Specifically, the server 104 can perform image channel fusion processing on the user interface test image and the user interface standard design image respectively, so as to fuse the information of the three RGB channels of the user interface test image and the user interface standard design image into a single brightness channel, and obtain the user interface test grayscale image and the user interface standard design grayscale image, thereby simplifying the image structure.
[0052] In a specific example, the image color space conversion function (cv2.cvtColor function) in the computer vision database (Open CV database) is used to perform image channel fusion processing on the user interface test image and the user interface standard design image respectively. The above is only a specific example. In actual applications, it can be flexibly set according to user needs and is not limited here.
[0053] Step 203, determining a first image structure similarity index and a first image pixel value variance according to the user interface test grayscale image, and determining a second image structure similarity index and a second image pixel value variance according to the user interface standard design grayscale image.
[0054] Specifically, the server 104 can determine a first image structure similarity index and a first image pixel value variance based on the user interface test grayscale image, and determine a second image structure similarity index and a second image pixel value variance based on the user interface standard design grayscale image, so that the structural difference between the user interface test grayscale image and the user interface standard design grayscale image can be understood through the first image structure similarity index and the second image structure similarity index, and the image pixel intensity difference between the user interface test grayscale image and the user interface standard design grayscale image can be understood through the first image pixel value variance and the second image pixel value variance.
[0055] In one embodiment, determining a first image structure similarity index and a first image pixel value variance according to a user interface test grayscale image, and determining a second image structure similarity index and a second image pixel value variance according to a user interface standard design grayscale image, comprises:
[0056] Based on the image structure similarity index calculation function of the image processing database, the first image structure similarity index is obtained after calculation according to the user interface test grayscale image, and the second image structure similarity index is obtained after calculation according to the user interface standard design grayscale image.
[0057] Specifically, server 104 calculates the image structure similarity index function based on the image processing database, obtains a first image structure similarity index after calculation according to the user interface test grayscale image, and obtains a second image structure similarity index after calculation according to the user interface standard design grayscale image. The first image structure similarity index and the second image structure similarity index can be used to test the similarity between the user interface test grayscale image and the user interface standard design grayscale image in terms of image brightness, image contrast and image structure, so as to accurately evaluate the similarity between the user interface test grayscale image and the user interface standard design grayscale image.
[0058] In a specific example, the image processing database may be but is not limited to a sc iki ti image database, and the image structure similarity index calculation function may be but is not limited to a compare_ss im function. The above are only specific examples and can be flexibly set according to user needs in actual applications and are not limited here.
[0059] In this embodiment, an image structure similarity index calculation function based on an image processing database is used to obtain a first image structure similarity index after calculation based on a user interface test grayscale image, and a second image structure similarity index is obtained after calculation based on a user interface standard design grayscale image. The first image structure similarity index and the second image structure similarity index can be used to test the similarity between the user interface test grayscale image and the user interface standard design grayscale image in terms of image brightness, image contrast and image structure, thereby facilitating accurate evaluation of the similarity between the user interface test grayscale image and the user interface standard design grayscale image.
[0060] In one embodiment, determining a first image structure similarity index and a first image pixel value variance according to a user interface test grayscale image, and determining a second image structure similarity index and a second image pixel value variance according to a user interface standard design grayscale image, comprises:
[0061] Calculate the variance of the pixel value of the first image according to the total number of pixels of the user interface test grayscale image, the brightness value of each pixel, and the average value of the brightness values of all pixels;
[0062] The total number of pixels of the grayscale image, the brightness value of each pixel and the average value of the brightness values of all pixels are calculated according to the user interface standard to obtain the pixel value variance of the second image.
[0063] Specifically, server 104 calculates the first image pixel value variance based on the total number of pixels of the user interface test grayscale image, the brightness value of each pixel and the average value of the brightness values of all pixels; then, calculates the second image pixel value variance based on the total number of pixels of the user interface standard design grayscale image, the brightness value of each pixel and the average value of the brightness values of all pixels, so as to understand the difference in image pixel intensity between the user interface test grayscale image and the user interface standard design grayscale image through the first image pixel value variance and the second image pixel value variance, and to accurately evaluate the similarity between the user interface test grayscale image and the user interface standard design grayscale image.
[0064] In a specific example, the first image pixel value variance is determined based on the following expression:
[0065]
[0066] Among them, C 1 is the variance of the pixel value of the first image; N is the total number of pixels in the user interface test grayscale image; I(i) is the brightness value of the pixel of the i-th user interface test grayscale image; u is the average brightness value of all pixels in the user interface test grayscale image. The above are only specific examples and can be flexibly set according to user needs in actual applications and are not limited here.
[0067] In this embodiment, a first image pixel value variance is calculated based on the total number of pixels of the user interface test grayscale image, the brightness value of each pixel and the average value of the brightness values of all pixels; then, a second image pixel value variance is calculated based on the total number of pixels of the user interface standard design grayscale image, the brightness value of each pixel and the average value of the brightness values of all pixels, so that the difference in image pixel intensity between the user interface test grayscale image and the user interface standard design grayscale image can be understood through the first image pixel value variance and the second image pixel value variance, so as to accurately evaluate the similarity between the user interface test grayscale image and the user interface standard design grayscale image.
[0068] Step 204 , performing calculations based on the first image structure similarity index, the second image structure similarity index, the first image pixel value variance, and the second image pixel value variance to obtain a comprehensive similarity score of the user interface test images.
[0069] The similarity comprehensive score is used to characterize the similarity between the user interface test image and the user interface standard design image. Specifically, the server 104 calculates the similarity comprehensive score of the user interface test image based on the first image structure similarity index, the second image structure similarity index, the first image pixel value variance, and the second image pixel value variance, so as to obtain the similarity comprehensive score of the user interface test image, thereby comprehensively evaluating the image similarity through the similarity comprehensive score, balancing the importance of different image characteristics, improving the test efficiency and test accuracy, and reducing the test cost.
[0070] In one embodiment, if Figure 4 As shown, the similarity comprehensive score of the user interface test image is obtained by calculation based on the first image structure similarity index, the second image structure similarity index, the first image pixel value variance and the second image pixel value variance, including steps 401 to 403.
[0071] Step 401: determine a similarity index ratio according to a quotient of a first image structure similarity index and a second image structure similarity index.
[0072] Step 402: determine a pixel value variance ratio according to a quotient of a first image pixel value variance and a second image pixel value variance.
[0073] Step 403, a calculation is performed based on the similarity index ratio, the weight of the similarity index ratio, the pixel value variance ratio and the weight of the pixel value variance ratio to obtain a comprehensive similarity score.
[0074] Specifically, a similarity index ratio is determined according to a quotient of a first image structure similarity index and a second image structure similarity index; then, a pixel value variance ratio is determined according to a quotient of a first image pixel value variance and a second image pixel value variance; finally, individual calculations are performed according to the similarity index ratio, the weight of the similarity index ratio, the pixel value variance ratio and the weight of the pixel value variance ratio to obtain a comprehensive similarity score, so that the image similarity can be comprehensively evaluated through the similarity comprehensive score, the importance of different image characteristics can be balanced, the test efficiency and test accuracy can be improved, and the test cost can be reduced.
[0075] In a specific example, the similarity comprehensive score of the user interface test image is determined based on the following expression:
[0076]
[0077] Among them, Z is the comprehensive score of similarity; S 1 is the first image structure similarity index; S 2 is the first image structure similarity index; a is the weight of the similarity index ratio; C 1 is the variance of the pixel value of the first image; C2 is the variance of the pixel value of the second image; b is the weight of the pixel value variance ratio. The above is only a specific example. In practical applications, it can be flexibly set according to user needs and is not limited here.
[0078] In this embodiment, a similarity index ratio is determined according to the quotient of the first image structure similarity index and the second image structure similarity index; then, a pixel value variance ratio is determined according to the quotient of the first image pixel value variance and the second image pixel value variance; finally, individual calculations are performed according to the similarity index ratio, the weight of the similarity index ratio, the pixel value variance ratio and the weight of the pixel value variance ratio to obtain a comprehensive similarity score, so that the image similarity can be comprehensively evaluated through the similarity comprehensive score, the importance of different image characteristics can be balanced, the test efficiency and test accuracy can be improved, and the test cost can be reduced.
[0079] The above-mentioned user interface testing method obtains a user interface test image and a user interface standard design image of an application; then, performs image channel fusion processing on the user interface test image and the user interface standard design image to obtain a user interface test grayscale image and a user interface standard design grayscale image; then, determines a first image structure similarity index and a first image pixel value variance according to the user interface test grayscale image, and determines a second image structure similarity index and a second image pixel value variance according to the user interface standard design grayscale image; finally, calculates based on the first image structure similarity index, the second image structure similarity index, the first image pixel value variance and the second image pixel value variance to obtain a similarity comprehensive score of the user interface test image; wherein the similarity comprehensive score is used to characterize the similarity between the user interface test image and the user interface standard design image, so that the image similarity can be comprehensively evaluated through the similarity comprehensive score, the importance of different image characteristics is balanced, the test efficiency and test accuracy are improved, and the test cost is reduced.
[0080] In one embodiment, if Figure 5 As shown, the method includes steps 501 to 503.
[0081] Step 501 , in response to the similarity comprehensive score being greater than a score threshold, image fusion processing is performed on the user interface test image and the user interface standard design image to obtain a user interface fusion image.
[0082] Specifically, when the server 104 recognizes that the comprehensive similarity score is greater than the score threshold, it indicates that the user interface test image and the user interface standard design image have significant differences, and then performs image fusion processing on the user interface test image and the user interface standard design image to obtain a user interface fusion image.
[0083] Step 502: Perform image processing on the user interface fusion image to obtain target difference regions.
[0084] The target difference region is a region where the user interface test image and the user interface standard design image are different. Specifically, the server 104 performs image processing on the user interface fusion image to obtain each target difference region.
[0085] In one embodiment, if Figure 6 As shown, image processing is performed on the user interface fusion image to obtain a target difference area, including steps 601 to 604.
[0086] Step 601: multiply the pixel value of each pixel point of the user interface fusion image by the maximum pixel value to obtain the pixel value of each processed pixel point.
[0087] Step 602 , converting the pixel value of each processed pixel point into an unsigned eight-bit integer data format to obtain the pixel value of each pixel point in a standard format.
[0088] Step 603 , based on the maximum inter-class variance method, the user interface fusion image is subjected to image binarization processing according to the pixel values of the pixel points in each standard format to obtain the user interface fusion image after the binarization processing.
[0089] Step 604 : determining target difference regions according to the black regions in the user interface fusion image after the binarization process.
[0090] Specifically, the server 104 multiplies the pixel value of each pixel of the user interface fusion image by the maximum pixel value to obtain the pixel value of each processed pixel; then, the pixel value of each processed pixel is converted into an unsigned eight-bit integer data format to obtain the pixel value of each pixel in a standard format, so as to facilitate subsequent threshold calculation and image binarization processing; then, based on the maximum between-class variance method (Ostu algorithm), the user interface fusion image is binarized according to the pixel values of the pixel points in each standard format to obtain the user interface fusion image after binarization processing; finally, each target difference area is determined according to the black area in the user interface fusion image after binarization processing, thereby improving the accuracy and efficiency of determining the target difference area.
[0091] In this embodiment, the pixel value of each pixel point of the user interface fusion image is multiplied by the maximum pixel value to obtain the pixel value of each processed pixel point; then, the pixel value of each processed pixel point is converted into an unsigned eight-bit integer data format to obtain the pixel value of each pixel point in a standard format, which is convenient for subsequent threshold calculation and image binarization processing; then, based on the maximum between-class variance method (Ostu algorithm), the user interface fusion image is subjected to image binarization processing according to the pixel values of each pixel point in the standard format to obtain the user interface fusion image after binarization processing; finally, each target difference area is determined according to the black area in the user interface fusion image after binarization processing, thereby improving the accuracy and efficiency of determining the target difference area.
[0092] Step 503 : annotate the user interface fusion image according to each target difference region to obtain an annotated user interface fusion image.
[0093] Specifically, the server 104 annotates the user interface fusion image according to each target difference area to obtain an annotated user interface fusion image, thereby facilitating the user to improve the user interface test image according to the annotated user interface fusion image, thereby improving the convenience of user interface testing and improving the visibility of image differences.
[0094] In one embodiment, if Figure 7 As shown, the user interface fusion image is annotated according to the target difference area to obtain the annotated user interface fusion image, including steps 701 to 703.
[0095] Step 701: Determine the contour of the corresponding target difference region according to each target difference region.
[0096] Step 702 : Calculate according to the outline of each target difference region to obtain the minimum circumscribed rectangle of the corresponding target difference region.
[0097] Step 703 : annotate the user interface fusion image according to each minimum bounding rectangle to obtain an annotated user interface fusion image.
[0098] Specifically, the contour of the corresponding target difference area is determined according to each target difference area; then, the minimum enclosing rectangle of the corresponding target difference area is calculated according to the contour of each target difference area; finally, the user interface fusion image is annotated according to each minimum enclosing rectangle to obtain the annotated user interface fusion image, which improves the convenience of user interface testing and the visibility of image differences.
[0099] In this embodiment, the contour of the corresponding target difference area is determined according to each target difference area; then, the minimum enclosing rectangle of the corresponding target difference area is calculated according to the contour of each target difference area; finally, the user interface fusion image is annotated according to each minimum enclosing rectangle to obtain the annotated user interface fusion image, which improves the convenience of user interface testing and improves the visibility of image differences.
[0100] In one embodiment, the method comprises:
[0101] In response to the similarity comprehensive score being less than or equal to the score threshold, the similarity comprehensive score is recorded and stored.
[0102] In one embodiment, the method comprises:
[0103] Generate a portable network graphics format file based on the annotated user interface fusion image.
[0104] Specifically, the server 104 generates a portable network graphics format file according to the annotated user interface fusion image, thereby improving the convenience of user interface testing.
[0105] In a specific example, generating a portable network graphics format file according to the annotated user interface fusion image includes:
[0106] The cv2.i mencode function is used to encode the annotated user interface fusion image to obtain a byte stream; then, the tofile algorithm is used to write the byte stream into a portable network graphics format file. The above is only a specific example. In actual applications, it can be flexibly set according to user needs and is not limited here.
[0107] In this embodiment, a portable network graphics format file is generated according to the annotated user interface fusion image, thereby improving the convenience of user interface testing.
[0108] It should be understood that although Figure 2-7 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-7 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0109] Second, as Figure 8 As shown, a user interface testing device is provided, which includes an image acquisition module 810, an image channel fusion module 820, a parameter calculation module 830 and a score calculation module 840.
[0110] Among them, the image acquisition module 810 is used to acquire the user interface test image and the user interface standard design image of the application; the image channel fusion module 820 is used to perform image channel fusion processing on the user interface test image and the user interface standard design image respectively, to obtain the user interface test grayscale image and the user interface standard design grayscale image; the parameter calculation module 830 is used to determine the first image structure similarity index and the first image pixel value variance according to the user interface test grayscale image, and determine the second image structure similarity index and the second image pixel value variance according to the user interface standard design grayscale image; the score calculation module 840 is used to calculate according to the first image structure similarity index, the second image structure similarity index, the first image pixel value variance and the second image pixel value variance, to obtain a comprehensive similarity score of the user interface test image; wherein the comprehensive similarity score is used to characterize the degree of similarity between the user interface test image and the user interface standard design image.
[0111] In one of the embodiments, the method includes an image annotation module.
[0112] Among them, the image annotation module is used to perform image fusion processing on the user interface test image and the user interface standard design image in response to the comprehensive similarity score being greater than the score threshold, so as to obtain the user interface fusion image; the image annotation module is used to perform image processing on the user interface fusion image to obtain each target difference area; wherein the target difference area is the area where there is a difference between the user interface test image and the user interface standard design image; the image annotation module is used to perform annotation processing on the user interface fusion image according to each target difference area, so as to obtain the annotated user interface fusion image.
[0113] In one of the embodiments, the image annotation module includes a target difference region determination unit.
[0114] Among them, the target difference area determination unit is used to multiply the pixel value of each pixel point of the user interface fusion image by the maximum pixel value to obtain the pixel value of each processed pixel point; the target difference area determination unit is used to convert the pixel value of each processed pixel point into an unsigned eight-bit integer data format to obtain the pixel value of each standard format pixel point; the target difference area determination unit is used to perform image binarization processing on the user interface fusion image according to the pixel values of each standard format pixel point based on the maximum inter-class variance method to obtain the user interface fusion image after binarization processing; the target difference area determination unit is used to determine each target difference area according to the black area in the user interface fusion image after binarization processing.
[0115] In one embodiment, the image annotation module includes an annotation processing unit.
[0116] Among them, the annotation processing unit is used to determine the outline of the corresponding target difference area according to each target difference area; the annotation processing unit is used to calculate according to the outline of each target difference area to obtain the minimum enclosing rectangle of the corresponding target difference area; the annotation processing unit is used to annotate the user interface fusion image according to each minimum enclosing rectangle to obtain the annotated user interface fusion image.
[0117] In one of the embodiments, the method includes a file generation module.
[0118] The file generation module is used to generate a portable network graphics format file according to the annotated user interface fusion image.
[0119] In one of the embodiments, the image acquisition module includes an image acquisition unit.
[0120] Among them, the image acquisition unit is used to determine the user interface uniform resource locator system address in response to the input operation of the user interface uniform resource locator system address of the application; the image acquisition unit is used to perform web page interface screenshot processing based on the user interface uniform resource locator system address based on the automated testing tool to obtain a user interface test image.
[0121] In one embodiment, the parameter calculation module 830 includes an image structure similarity index calculation unit.
[0122] Among them, the image structure similarity index calculation unit is used to calculate the image structure similarity index calculation function based on the image processing database, obtain the first image structure similarity index after calculating according to the user interface test grayscale image, and obtain the second image structure similarity index after calculating according to the user interface standard design grayscale image.
[0123] In one embodiment, the parameter calculation module 830 includes an image pixel value variance calculation unit.
[0124] Among them, the image pixel value variance calculation unit is used to calculate the first image pixel value variance according to the total number of pixels of the user interface test grayscale image, the brightness value of each pixel and the average value of the brightness values of all pixels; the image pixel value variance calculation unit is used to calculate the second image pixel value variance according to the total number of pixels of the user interface standard design grayscale image, the brightness value of each pixel and the average value of the brightness values of all pixels.
[0125] In one embodiment, the score calculation module 840 includes a score calculation unit.
[0126] Among them, the scoring calculation unit is used to determine the similarity index ratio according to the quotient of the first image structure similarity index and the second image structure similarity index; the scoring calculation unit is used to determine the pixel value variance ratio according to the quotient of the first image pixel value variance and the second image pixel value variance; the scoring calculation unit is used to perform individual calculations based on the similarity index ratio, the weight of the similarity index ratio, the pixel value variance ratio and the weight of the pixel value variance ratio to obtain a comprehensive similarity score.
[0127] For the specific definition of the user interface testing device, please refer to the definition of the user interface testing method above, which will not be repeated here. Each module in the above-mentioned user interface testing device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0128] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store user interface test images and user interface standard design images. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a user interface testing method is implemented.
[0129] Those skilled in the art will understand that Fig. 9The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0130] In a third aspect, a computer device is provided. The computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps of any method in the above method embodiments are implemented.
[0131] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any method in the above method embodiments are implemented.
[0132] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.
[0133] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0134] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A user interface testing method, the method comprising: Obtaining a user interface test image and a user interface standard design image of an application; Performing image channel fusion processing on the user interface test image and the user interface standard design image respectively to obtain a user interface test grayscale image and a user interface standard design grayscale image; Determining a first image structure similarity index and a first image pixel value variance according to the user interface test grayscale image, and determining a second image structure similarity index and a second image pixel value variance according to the user interface standard design grayscale image; A calculation is performed based on the first image structure similarity index, the second image structure similarity index, the first image pixel value variance, and the second image pixel value variance to obtain a comprehensive similarity score of the user interface test image; wherein the comprehensive similarity score is used to characterize the degree of similarity between the user interface test image and the user interface standard design image.
2. The method according to claim 1, characterized in that The method further comprises: In response to the similarity comprehensive score being greater than a score threshold, performing image fusion processing on the user interface test image and the user interface standard design image to obtain a user interface fusion image; Performing image processing on the user interface fusion image to obtain target difference regions; wherein the target difference regions are regions where differences exist between the user interface test image and the user interface standard design image; The user interface fusion image is annotated according to each of the target difference regions to obtain the annotated user interface fusion image.
3. The method according to claim 2, characterized in that The performing image processing on the user interface fusion image to obtain a target difference area includes: Multiplying the pixel value of each pixel point of the user interface fusion image by the maximum pixel value to obtain the pixel value of each pixel point after processing; Convert the pixel values of each of the processed pixel points into an unsigned eight-bit integer data format to obtain the pixel values of each of the pixel points in a standard format; Based on the maximum inter-class variance method, performing image binarization processing on the user interface fusion image according to the pixel values of the pixel points in each of the standard formats to obtain the user interface fusion image after binarization processing; The target difference regions are determined according to the black regions in the user interface fusion image after the binarization process.
4. The method according to claim 2, characterized in that: The step of labeling the user interface fusion image according to the target difference area to obtain the labeled user interface fusion image includes: Determine the contour of the corresponding target difference area according to each target difference area; Calculating according to the contours of each target difference region to obtain the minimum circumscribed rectangle of the corresponding target difference region; The user interface fusion image is annotated according to each of the minimum circumscribed rectangles to obtain the annotated user interface fusion image.
5. The method according to claim 1, characterized in that The step of obtaining a user interface test image of an application includes: In response to an input operation on a user interface uniform resource locator system address of the application, determining the user interface uniform resource locator system address; Based on the automated testing tool, a web page interface screenshot is processed according to the user interface uniform resource locator system address to obtain the user interface test image.
6. The method according to claim 1, characterized in that Determining a first image structure similarity index and a first image pixel value variance according to the user interface test grayscale image, and determining a second image structure similarity index and a second image pixel value variance according to the user interface standard design grayscale image, comprises: An image structure similarity index calculation function based on an image processing database is used to calculate the first image structure similarity index according to the user interface test grayscale image, and to calculate the second image structure similarity index according to the user interface standard design grayscale image; Calculate the pixel value variance of the first image according to the total number of pixels of the user interface test grayscale image, the brightness value of each pixel, and the average value of the brightness values of all pixels; The total number of pixels of the grayscale image, the brightness value of each pixel and the average value of the brightness values of all pixels are calculated according to the user interface standard design to obtain the pixel value variance of the second image.
7. The method according to claim 1, characterized in that The calculating according to the first image structure similarity index, the second image structure similarity index, the first image pixel value variance and the second image pixel value variance to obtain the similarity comprehensive score of the user interface test image includes: determining a similarity index ratio according to a quotient of the first image structure similarity index and the second image structure similarity index; Determining a pixel value variance ratio according to a quotient of the first image pixel value variance and the second image pixel value variance; A calculation is performed based on the similarity index ratio, the weight of the similarity index ratio, the pixel value variance ratio and the weight of the pixel value variance ratio to obtain the similarity comprehensive score.
8. A user interface testing device, characterized in that: The device comprises: An image acquisition module, used to acquire a user interface test image and a user interface standard design image of an application; An image channel fusion module, used to perform image channel fusion processing on the user interface test image and the user interface standard design image respectively, to obtain a user interface test grayscale image and a user interface standard design grayscale image; a parameter calculation module, for determining a first image structure similarity index and a first image pixel value variance according to the user interface test grayscale image, and determining a second image structure similarity index and a second image pixel value variance according to the user interface standard design grayscale image; A scoring calculation module is used to calculate based on the first image structure similarity index, the second image structure similarity index, the first image pixel value variance and the second image pixel value variance to obtain a comprehensive similarity score of the user interface test image; wherein the comprehensive similarity score is used to characterize the degree of similarity between the user interface test image and the user interface standard design image.
9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
Citation Information
Cited By
Simulation evaluation method for automatic testing equipment of vehicle machine
CN120216356A
