Image-based user interface testing method, device, equipment and storage medium

Abstract

Embodiments of the present application provide a kind of based on image user interface test method, device, equipment and storage medium.The method comprises: receiving the snapshot timing data sent by user equipment, and the user interface UI design image of user equipment is acquired;Wherein, UI design image is the UI image of original design user equipment;Snapshot timing data includes at least one time under the UI snapshot of user equipment;From snapshot timing data, the similarity of UI snapshot between UI design image is highest, is the similar UI snapshot of UI design image;According to the image feature of UI design image and the image feature of similar UI snapshot, the UI test result of user equipment is generated;Wherein, UI test result is the UI restoration degree test result of user equipment.The method is used to reach the effect of improving the efficiency and accuracy of UI test.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to an image-based user interface testing method, apparatus, device, and storage medium. Background Technology

[0002] After the software product development is completed, the user interface (UI) needs to be tested for UI fidelity to ensure that the actual interface is visually consistent with the original design interface, so as to control product quality and user experience.

[0003] In existing technologies, testers manually compare the actual interface with the original design interface to find differences in order to complete the UI fidelity test.

[0004] However, the above methods, which use manual comparison, require repeated manual comparison operations, resulting in a decrease in the efficiency and accuracy of UI testing. Summary of the Invention

[0005] This application provides an image-based user interface testing method, apparatus, device, and storage medium to improve the efficiency and accuracy of UI testing.

[0006] In a first aspect, embodiments of this application provide an image-based user interface testing method, the method being applied to a server; the method includes:

[0007] The system receives snapshot timing data sent by a user equipment and acquires a user interface (UI) design image of the user equipment; wherein the UI design image is the original design of the user equipment's UI image; the snapshot timing data includes at least one UI snapshot of the user equipment at any given moment.

[0008] The UI snapshot with the highest similarity to the UI design image is determined from the snapshot time series data and is taken as the similar UI snapshot to the UI design image;

[0009] Based on the image features of the UI design image and the image features of the similar UI snapshot, a UI test result for the user device is generated; wherein, the UI test result is a UI fidelity test result for the user device.

[0010] In one possible implementation, the UI test results of the user device are generated based on the image features of the UI design image and the image features of the similar UI snapshot, including:

[0011] A first element feature is determined from the image features of the UI design image; and a second element feature is determined from the image features of the similar UI snapshot; wherein, the first element feature is the element feature of each page element in the UI design image; and the second element feature is the element feature of each page element in the similar UI snapshot;

[0012] The UI test results of the user device are generated based on the first element feature and the second element feature.

[0013] In one possible implementation, determining a first element feature among the image features of the UI design image; and determining a second element feature among the image features of the similar UI snapshot, including:

[0014] The UI design image and the similar UI snapshot are respectively subjected to image segmentation processing to obtain at least one first graphical user interface (GUI) element image corresponding to the UI design image and at least one second GUI element image corresponding to the similar UI snapshot;

[0015] The first GUI element image is processed to obtain the first element feature; and the second GUI element image is processed to obtain the second element feature.

[0016] In one possible implementation, image segmentation processing is performed on the UI design image and the similar UI snapshot respectively to obtain at least one first graphical user interface (GUI) element image corresponding to the UI design image and at least one second GUI element image corresponding to the similar UI snapshot, including:

[0017] Edge detection is performed on the UI design image and the similar UI snapshot respectively to obtain the original contour information of the UI design image and the snapshot contour information of the similar UI snapshot; wherein, the original contour information includes at least one design contour image; the design contour image represents the background contour of the UI element in the UI design image; the snapshot contour information includes at least one snapshot contour image; the snapshot contour image represents the background contour of the UI element in the similar UI snapshot;

[0018] The design outline image is segmented to obtain the first GUI element image; and the snapshot outline image is segmented to obtain the second GUI element image.

[0019] In one possible implementation, the method further includes:

[0020] The design outline image and the snapshot outline image are subjected to a flood filling process to obtain the processed design outline image and the processed snapshot outline image.

[0021] In one possible implementation, feature processing is performed on each of the first GUI element images to obtain the first element features; and feature processing is performed on each of the second GUI element images to obtain the second element features, including:

[0022] The first GUI element image and the second GUI element image are subjected to target detection processing respectively to obtain first component data and second component data; wherein, the first component data represents the foreground component in the first GUI element image; and the second component data represents the foreground component in the second GUI element image.

[0023] Attribute extraction processing is performed on the foreground component data in each of the first GUI element images to obtain the component attribute features in the first element features; and attribute extraction processing is performed on the foreground component data in each of the second GUI element images to obtain the component attribute features in the second element features;

[0024] Based on the coordinates of the cutting points of each of the first GUI element images, the element position features in the first element features are determined; and based on the coordinates of the cutting points of each of the second GUI element images, the element position features in the second element features are determined.

[0025] In one possible implementation, generating the UI test results of the user device based on the first element feature and the second element feature includes:

[0026] Based on the first element features, a first feature tree corresponding to the UI design image is generated; and based on the second element features, a second feature tree corresponding to the similar UI snapshot is generated; wherein, the first feature tree represents the element hierarchy of each UI element in the UI design image and the inclusion relationship between each UI element; the second feature tree represents the hierarchy, attributes, and inclusion relationship between each UI element in the similar UI snapshot;

[0027] The first feature tree and the second feature tree are traversed and compared to obtain the UI test results of the user device.

[0028] In one possible implementation, determining the UI snapshot with the highest similarity to the UI design image from the snapshot time-series data, as the similar UI snapshot to the UI design image, includes:

[0029] Determine the image vector corresponding to the UI design image; and determine the snapshot vector set corresponding to the snapshot time series data; wherein, the snapshot vector set includes the snapshot vector corresponding to each UI snapshot;

[0030] Determine the cosine similarity between the image vector and the snapshot vector;

[0031] The UI snapshot corresponding to the snapshot vector whose cosine similarity value is closest to 1 is determined as the similar UI snapshot.

[0032] In one possible implementation, the image vector corresponding to the UI design image is determined; and the set of snapshot vectors corresponding to the snapshot time series data is determined, including:

[0033] A first three-channel histogram corresponding to the UI design image is determined, and a second three-channel histogram corresponding to the UI snapshot is determined; wherein, the first three-channel histogram includes first color information of each first pixel in the UI design image, and the first color information includes the color value of the first pixel in each preset color channel; the second three-channel histogram includes second color information of each second pixel in the UI snapshot, and the second color information includes the color value of the second pixel in each preset color channel;

[0034] Based on the first three-channel histogram, the image vector corresponding to the UI design image is determined; and based on the second three-channel histogram, the snapshot vector corresponding to the UI snapshot is determined.

[0035] In one possible implementation, the image vector corresponding to the UI design image is determined based on the first three-channel histogram; and the snapshot vector corresponding to the UI snapshot is determined based on the second three-channel histogram, including:

[0036] At least one preset color value corresponding to each preset color channel is partitioned and combined to obtain at least one combined partition; wherein, the combined partition includes a color partition corresponding to each preset color channel; the color partition includes a preset number of consecutive preset color values ​​under each preset color channel; the preset color values ​​in each color partition corresponding to the preset color channel are not repeated.

[0037] Based on the first color information, a first combined partition corresponding to the first pixel is determined from each of the combined partitions; wherein, the first combined partition is a combined partition in which the color partition under each preset color channel is consistent with the first color partition corresponding to the first pixel under the preset color channel; the first color partition is the color partition in which the preset color value is consistent with the color value under the preset color channel in the first color information is located.

[0038] Based on the second color information, a second combined partition corresponding to the second pixel is determined from each of the combined partitions; wherein, the second combined partition is a combined partition in which the color partition under each preset color channel is consistent with the second color partition corresponding to the second pixel under the preset color channel; the second partition is the color partition in which the preset color value is consistent with the color value under the preset color channel in the second color information is located.

[0039] The first number corresponding to each of the combined partitions is concatenated to obtain the image vector corresponding to the UI design image; and for each UI snapshot, the second number corresponding to each of the combined partitions is concatenated to obtain the snapshot vector corresponding to the UI snapshot; wherein, the first number is the total number of all first pixels corresponding to the combined partition; and the second number is the total number of all second pixels corresponding to each combined partition in each UI snapshot.

[0040] Secondly, embodiments of this application provide an image-based user interface testing method, the method being applied to a user device; the method includes:

[0041] Create an initial drawing element, and draw each page element of the user interface (UI) of the user device at the current moment onto the initial drawing element to obtain the drawing element;

[0042] The drawing elements are processed to export data, generating a UI snapshot of the user device at the current moment; and the UI snapshot is added to the snapshot time series data.

[0043] The snapshot time-series data is sent to a server; wherein the server is used to perform the method as described in the first aspect.

[0044] In one possible implementation, each page element in the user interface (UI) of the user device at the current moment is drawn onto the initial drawing element to obtain the drawing element, including:

[0045] The page currently displayed on the user device is parsed and processed using a preset Document Object Model (DOM) tree to obtain an element set; wherein, the element set includes the style and layout information of each page element in the user interface (UI) of the user device;

[0046] Based on the DOM tree and the element set, each page element in the element set is drawn onto the initial drawing element to obtain the drawing element.

[0047] In one possible implementation, the method further includes:

[0048] If it is determined that the number of snapshots in the snapshot time series data is greater than or equal to a preset number, then the document object parsing process for the user interface (UI) of the user device is stopped; wherein, the number of snapshots is the total number of UI snapshots in the snapshot time series data.

[0049] Thirdly, embodiments of this application provide an image-based user interface testing device, the device being applied to a server; the device includes:

[0050] The acquisition module is used to receive snapshot time-series data sent by the user equipment and acquire the design user interface (UI) image of the user equipment; wherein the UI design image is the original design UI image of the user equipment; the snapshot time-series data includes at least one UI snapshot of the user equipment at a given moment;

[0051] The determination module is used to determine the UI snapshot with the highest similarity to the UI design image from the snapshot time series data, and to identify the similar UI snapshot to the UI design image.

[0052] The generation module is used to generate the UI test results of the user device based on the image features of the UI design image and the image features of the similar UI snapshot; wherein, the UI test results are the UI fidelity test results of the user device.

[0053] In one possible implementation, the generation module is specifically configured to: determine a first element feature in the image features of the UI design image; and determine a second element feature in the image features of the similar UI snapshot; wherein the first element feature is the element feature of each page element in the UI design image; the second element feature is the element feature of each page element in the similar UI snapshot; and generate the UI test result of the user device based on the first element feature and the second element feature.

[0054] In one possible implementation, the generation module is specifically configured to: perform image segmentation processing on the UI design image and the similar UI snapshot respectively to obtain at least one first graphical user interface (GUI) element image corresponding to the UI design image and at least one second GUI element image corresponding to the similar UI snapshot; perform feature processing on each first GUI element image to obtain first element features; and perform feature processing on each second GUI element image to obtain second element features.

[0055] In one possible implementation, the generation module is specifically configured to: perform edge detection on the UI design image and the similar UI snapshot respectively to obtain the original contour information of the UI design image and the snapshot contour information of the similar UI snapshot; wherein, the original contour information includes at least one design contour image; the design contour image represents the background contour of UI elements in the UI design image; the snapshot contour information includes at least one snapshot contour image; the snapshot contour image represents the background contour of UI elements in the similar UI snapshot; perform segmentation processing on the design contour image to obtain the first GUI element image; and perform segmentation processing on the snapshot contour image to obtain the second GUI element image.

[0056] In one possible implementation, the generation module is further specifically used to: perform a flood filling process on the design outline image and the snapshot outline image to obtain a processed design outline image and a processed snapshot outline image.

[0057] In one possible implementation, the generation module is further specifically configured to: perform target detection processing on the first GUI element image and the second GUI element image respectively to obtain first component data and second component data; wherein, the first component data represents the foreground component in the first GUI element image; the second component data represents the foreground component in the second GUI element image; perform attribute extraction processing on the foreground component data in each of the first GUI element images to obtain component attribute features in the first element features; and perform attribute extraction processing on the foreground component data in each of the second GUI element images to obtain component attribute features in the second element features; determine the element position features in the first element features based on the cut point coordinates of each of the first GUI element images; and determine the element position features in the second element features based on the cut point coordinates of each of the second GUI element images.

[0058] In one possible implementation, the generation module is further configured to: generate a first feature tree corresponding to the UI design image based on the first element features; and generate a second feature tree corresponding to the similar UI snapshot based on the second element features; wherein the first feature tree represents the element hierarchy of each UI element in the UI design image and the inclusion relationship between each UI element; the second feature tree represents the hierarchy, attributes, and inclusion relationship between each UI element in the similar UI snapshot; and perform traversal and comparison processing on the first feature tree and the second feature tree to obtain the UI test result of the user device.

[0059] In one possible implementation, the determining module is specifically used to: determine the image vector corresponding to the UI design image; and determine the snapshot vector set corresponding to the snapshot time series data; wherein the snapshot vector set includes the snapshot vector corresponding to each UI snapshot; determine the cosine similarity between the image vector and the snapshot vector; and determine the UI snapshot corresponding to the snapshot vector whose cosine similarity value is closest to 1, which is the similar UI snapshot.

[0060] In one possible implementation, the determining module is specifically configured to: determine a first three-channel histogram corresponding to the UI design image, and determine a second three-channel histogram corresponding to the UI snapshot; wherein, the first three-channel histogram includes first color information of each first pixel in the UI design image, the first color information including the color value of the first pixel in each preset color channel; the second three-channel histogram includes second color information of each second pixel in the UI snapshot, the second color information including the color value of the second pixel in each preset color channel; determine an image vector corresponding to the UI design image based on the first three-channel histogram; and determine a snapshot vector corresponding to the UI snapshot based on the second three-channel histogram.

[0061] In one possible implementation, the determining module is specifically configured to: perform partitioning and combination processing on at least one preset color value corresponding to each preset color channel to obtain at least one combined partition; wherein, the combined partition includes a color partition corresponding to each preset color channel; the color partition includes a preset number of consecutive preset color values ​​under each preset color channel; the preset color values ​​in each color partition corresponding to the preset color channel are mutually exclusive; and, based on the first color information, determine the first combined partition corresponding to the first pixel from each of the combined partitions; wherein, the first combined partition is a combined partition in which both the color partition under each preset color channel and the first color partition corresponding to the first pixel under that preset color channel are consistent; the first color partition is the location of the preset color value that is consistent with the color value under the preset color channel in the first color information. Color partitioning; based on the second color information, determine the second combined partition corresponding to the second pixel from each of the combined partitions; wherein, the second combined partition is a combined partition in which the color partition under each preset color channel is consistent with the second color partition corresponding to the second pixel under the preset color channel; the second color partition is the color partition where the preset color value is consistent with the color value under the preset color channel in the second color information is located; concatenate the first number corresponding to each of the combined partitions to obtain the image vector corresponding to the UI design image; and for each UI snapshot, concatenate the second number corresponding to each of the combined partitions to obtain the snapshot vector corresponding to the UI snapshot; wherein, the first number is the total number of all first pixels corresponding to the combined partition; the second number is the total number of all second pixels corresponding to each combined partition in each UI snapshot.

[0062] Fourthly, embodiments of this application provide an image-based user interface testing apparatus, which is applied to a user equipment; the apparatus includes:

[0063] A drawing module is used to create initial drawing elements and draw each page element in the user interface (UI) of the user device at the current moment onto the initial drawing elements to obtain drawing elements;

[0064] The generation module is used to export data from the drawing elements, generate a UI snapshot of the user device at the current moment, and add the UI snapshot to the snapshot time series data.

[0065] A sending module is used to send the snapshot time-series data to a server; wherein the server includes the apparatus described in the third aspect.

[0066] In one possible implementation, the drawing module is specifically used to: parse and process the page currently displayed by the user device using a preset Document Object Model (DOM) tree to obtain an element set; wherein the element set includes the style and layout information of each page element in the user interface (UI) of the user device; and draw each page element in the element set onto the initial drawing element according to the DOM tree and the element set to obtain the drawing element.

[0067] In one possible implementation, the device is further configured to: if it is determined that the number of snapshots in the snapshot time series data is greater than or equal to a preset number, then stop performing document object parsing processing on the user interface (UI) of the user device; wherein the number of snapshots is the total number of UI snapshots in the snapshot time series data.

[0068] Fifthly, embodiments of this application provide a server, including: a memory and a processor;

[0069] The memory stores computer-executed instructions;

[0070] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0071] Sixthly, embodiments of this application provide a user equipment, including: a memory and a processor;

[0072] The memory stores computer-executed instructions;

[0073] The processor executes computer execution instructions stored in the memory, causing the processor to perform the second aspect and / or various possible implementations of the second aspect as described above.

[0074] In a seventh aspect, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods of the first and / or second aspects described above.

[0075] Eighthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the methods of the first and / or second aspects described above.

[0076] The image-based user interface testing method provided in this application receives snapshot time-series data sent by a user device and acquires a designed user interface (UI) image of the user device. The UI design image is the original UI design image of the user device. The snapshot time-series data includes at least one UI snapshot of the user device at a given moment. The method identifies the UI snapshot with the highest similarity to the UI design image from the snapshot time-series data, defining it as a similar UI snapshot. Based on the image features of the UI design image and the image features of the similar UI snapshots, a UI test result for the user device is generated. This UI test result is the UI fidelity test result of the user device. By acquiring multiple UI snapshots from the user device and selecting the UI snapshot most similar to the original UI design image, the method compares the image features of this most similar UI snapshot with the image features of the UI design image to obtain the automatic UI fidelity test result, thereby improving the efficiency and accuracy of UI testing. Attached Figure Description

[0077] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0078] Figure 1 This application provides an illustration of an application scenario.

[0079] Figure 2 A flowchart illustrating an image-based user interface testing method provided in this application embodiment;

[0080] Figure 3 A flowchart illustrating a user interface testing scheme provided in an embodiment of this application;

[0081] Figure 4 A flowchart illustrating another image-based user interface testing method provided in this application embodiment;

[0082] Figure 5 A flowchart illustrating yet another image-based user interface testing method provided in this application embodiment;

[0083] Figure 6 A flowchart illustrating a user interface testing scheme provided in an embodiment of this application;

[0084] Figure 7 A schematic diagram illustrating a DSL JSON example provided in an embodiment of this application;

[0085] Figure 8 A schematic diagram illustrating a UI fidelity test scenario provided in an embodiment of this application;

[0086] Figure 9A schematic diagram of the structure of an image-based user interface testing device provided in an embodiment of this application;

[0087] Figure 10 A schematic diagram of another image-based user interface testing device provided in an embodiment of this application;

[0088] Figure 11 This application provides a schematic diagram of the structure of a server according to an embodiment of the present application.

[0089] Figure 12 This is a schematic diagram of the structure of a user equipment provided in an embodiment of this application.

[0090] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0091] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0092] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, they do not violate public order and good morals, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0093] Furthermore, the technical solution involved in this application, which involves big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.) and the use of artificial intelligence technology for automated decision-making, and makes decisions that have a significant impact on personal rights based on the results of automated decision-making, provides users with corresponding operation entry points for users to choose to agree to or reject the results of automated decision-making; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0094] It should be noted that this application can be used in the field of artificial intelligence, or in any field other than artificial intelligence; the application field of this application is not limited.

[0095] Figure 1 This application provides an illustration of an application scenario, such as... Figure 1 As shown, the specific application scenario of this application is as follows: After the software product is developed, it is necessary to conduct a UI fidelity test on the user interface (UI) to ensure that the actual UI interface is visually consistent with the original design UI interface, so as to control product quality and user experience.

[0096] Based on the above scenarios, it can be seen that in the existing technology, the UI fidelity test is completed by having testers manually compare the actual interface with the original design interface to find differences. However, this method requires repeated manual comparison, which reduces the efficiency and accuracy of UI testing.

[0097] The image-based user interface testing method provided in this application obtains multiple UI snapshots from the user device, selects the UI snapshot that is most similar to the UI image of the user device in the original design, compares the image features of the most similar UI snapshot with the image features of the UI design image, and obtains the result of automatic UI fidelity testing, thus solving the technical problem of reduced efficiency and accuracy of UI testing.

[0098] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0099] Figure 2 A flowchart illustrating an image-based user interface testing method provided in this application embodiment is shown below. Figure 2 As shown, the method includes:

[0100] 201. Receive snapshot timing data sent by the user equipment and obtain the user interface design image (UI) of the user equipment; wherein, the UI design image is the original design of the user equipment's UI image; the snapshot timing data includes at least one UI snapshot of the user equipment at any given moment.

[0101] For example, the execution subject of this embodiment may be a server. Figure 3 A flowchart illustrating a user interface testing scheme provided in an embodiment of this application is shown below. Figure 3As shown, in a real-world web application runtime environment, a user device, based on a UI snapshot generation plugin, can collect UI snapshots (C1, C2, ...) of the webpage displayed on the user device at one or more moments within the current time period. These snapshots are then packaged into snapshot time-series data and sent to the server. The server-based algorithm module receives the snapshot time-series data sent by the user device and obtains the UI image of the webpage displayed on the user device in the original UI design draft—the UI design image—for further processing.

[0102] 202. Identify the UI snapshot with the highest similarity to the UI design image from the snapshot time series data, and identify it as the similar UI snapshot to the UI design image.

[0103] For example, combined Figure 3 The server uses an image detection algorithm based on the pre-matching module to select the UI snapshot that best matches and has the highest similarity to the UI design image from the UI snapshot time series data, and then selects the similar UI snapshot for that UI design image.

[0104] 203. Based on the image features of the UI design image and the image features of similar UI snapshots, generate the UI test results for the user device; where the UI test results are the UI fidelity test results for the user device.

[0105] For example, combined Figure 3 The server uses a feature comparison algorithm based on the precise comparison module to extract image features of the UI design image and image features of similar UI snapshots, such as key features like the position, size, color, and rounded corners of each part. The extracted image features of the UI design image and the image features of similar UI snapshots are then compared to obtain the UI fidelity test results for the user device, i.e., the UI test results, such as whether there are obvious differences.

[0106] This embodiment provides an image-based user interface testing method. By acquiring multiple UI snapshots from a user device and selecting the UI snapshot that is most similar to the UI image of the user device in the original design, the image features of the most similar UI snapshot are compared with the image features of the UI design image to obtain the result of automatic UI fidelity testing, thereby improving the efficiency and accuracy of UI testing.

[0107] Figure 4 A flowchart illustrating another image-based user interface testing method provided in this application embodiment is shown below. Figure 4 As shown, the method includes:

[0108] 301. The user equipment creates an initial drawing element and draws each page element in the user interface (UI) at the current moment onto the initial drawing element to obtain the drawing element.

[0109] For example, based on the user device, an initial drawing element, such as a canvas element, is created, and each page element in the user interface (UI) of the user device at the current moment is obtained. Then, using canvas technology, all page elements are drawn onto the canvas element, thereby obtaining the drawn drawing element.

[0110] 302. The user equipment performs data export processing on the drawing elements to generate a UI snapshot of the user equipment at the current moment; and adds the UI snapshot to the snapshot time series data.

[0111] For example, based on the snapshot generation plugin of the user device, the drawn cavans elements are exported as image data to obtain the UI snapshot of the user device at the current moment, and the UI snapshot is added to the snapshot time series data.

[0112] 303. The user equipment sends snapshot time-series data to the server; wherein the server is used to perform the aforementioned method for testing the image-based user interface.

[0113] For example, the user device sends all UI snapshots from the generated snapshot time-series data to the server, so that the server performs a fidelity test on all UI snapshots based on the algorithm module.

[0114] 304. Receive snapshot timing data sent by the user equipment and acquire the user interface (UI) design image of the user equipment; wherein, the UI design image is the original design of the user equipment's UI image; the snapshot timing data includes at least one UI snapshot of the user equipment at any given moment.

[0115] For example, this step can be referred to as step 201, which will not be repeated here.

[0116] 305. The server determines the UI snapshot with the highest similarity to the UI design image from the snapshot time series data, and identifies it as the similar UI snapshot to the UI design image.

[0117] For example, this step can be referred to as step 202, which will not be repeated here.

[0118] 306. The server generates UI test results for the user device based on the image features of the UI design image and the image features of similar UI snapshots; wherein, the UI test results are the UI fidelity test results for the user device.

[0119] For example, this step can be referred to as step 203, which will not be repeated here.

[0120] In this embodiment, based on the above embodiments, on the one hand, the user device completes the snapshot collection of the page in the real running environment of the web application with the help of canvas technology. The obtained reproduction test results can include the influence of the device model, thereby improving the accuracy of the UI reproduction test results.

[0121] Figure 5 A flowchart illustrating another image-based user interface testing method provided in this application embodiment is shown below. Figure 5 As shown, the method includes:

[0122] 401. The user device creates initial drawing elements and parses the page currently displayed by the user device through a preset Document Object Model (DOM) tree to obtain an element set; wherein the element set includes the style and layout information of each page element in the user device's user interface (UI).

[0123] For example, based on the user device, an initial drawing element, such as a canvas element, is created. Based on the UI snapshot generation plugin, the DOM tree of the user device's page can be parsed every 500ms (the time can be customized) using a preset document object model. The DOM tree is then used to calculate an element set, which includes the style and layout information of each element in the user device's UI, such as hierarchy, size, and color.

[0124] 402. The user device draws each page element in the element set onto the initial drawing element based on the DOM tree and the element set to obtain the drawing element.

[0125] For example, the user device uses canvas technology to draw each page element onto the canvas element according to the DOM tree structure and the style information of each page element in the element set, thus obtaining the drawn canvas element, i.e., the drawing element.

[0126] 403. The user device performs data export processing on the drawing elements to generate a UI snapshot of the user device at the current moment; and adds the UI snapshot to the snapshot time series data.

[0127] For example, this step can be referred to as step 302, which will not be repeated here.

[0128] One example also includes: if the number of snapshots in the snapshot time series data is determined to be greater than or equal to a preset number, then stop document object parsing processing of the user interface (UI) of the user device; wherein, the number of snapshots is the total number of UI snapshots in the snapshot time series data.

[0129] For example, when the total number of UI snapshots in the snapshot time-series data collected by the user device's page, i.e., the number of snapshots, is greater than or equal to a preset number, the document object parsing process for the user device's UI is stopped.

[0130] For example, the generation of UI snapshot data for the current page ends when 10 UI snapshots (the number can be customized) are collected from the user's device page.

[0131] 404. The user equipment sends snapshot timing data to the server; wherein the server is used to perform the aforementioned method for testing the image-based user interface.

[0132] For example, this step can be referred to as step 303, which will not be repeated here.

[0133] 405. The server receives snapshot timing data sent by the user device and obtains the user interface (UI) design image of the user device; wherein, the UI design image is the original design of the user device's UI image; the snapshot timing data includes at least one UI snapshot of the user device at any given moment.

[0134] For example, this step can be referred to as step 201, which will not be repeated here.

[0135] 406. The server determines the image vector corresponding to the UI design image; and determines the snapshot vector set corresponding to the snapshot time series data; wherein the snapshot vector set includes the snapshot vector corresponding to each UI snapshot.

[0136] For example, the server can encode the UI design image using a preset vector encoder to obtain the image vector corresponding to the UI design image; and based on the preset vector encoder, encode each UI snapshot in the snapshot time series data to obtain the snapshot vector corresponding to each UI snapshot in the snapshot vector set.

[0137] In one example, step 406 includes:

[0138] The first step is to determine the first three-channel histogram corresponding to the UI design image and the second three-channel histogram corresponding to the UI snapshot. The first three-channel histogram includes the first color information of each first pixel in the UI design image, and the first color information includes the color value of the first pixel in each preset color channel. The second three-channel histogram includes the second color information of each second pixel in the UI snapshot, and the second color information includes the color value of the second pixel in each preset color channel.

[0139] The second step is to determine the image vector corresponding to the UI design image based on the first three-channel histogram, and to determine the snapshot vector corresponding to the UI snapshot based on the second three-channel histogram.

[0140] For example, during vector generation, the server performs color recognition processing on the UI design image based on the image color processing module to obtain a first three-channel histogram corresponding to the UI design image. This first three-channel histogram includes the color value of each first pixel in the UI design image under each preset color channel, i.e., first color information. Simultaneously, based on the image color processing module, it performs color recognition processing on each UI snapshot to obtain a second three-channel histogram corresponding to each UI snapshot. Each second three-channel histogram includes the color value of each second pixel in each UI snapshot under each preset color channel, i.e., second color information. The preset color channels are either red, green, or blue channels. Based on the server's image vector encoder, the first color information of each first pixel in the first three-channel histogram corresponding to the UI design image is vector-encoded to obtain the image vector corresponding to the UI design image. Simultaneously, based on the image vector encoder, the second color information of each second pixel in the second three-channel histogram corresponding to each UI snapshot is vector-encoded to obtain the snapshot vector corresponding to each UI snapshot.

[0141] In one example, the first step includes:

[0142] Step 1: Perform partitioning and combination processing on at least one preset color value corresponding to each preset color channel to obtain at least one combined partition; wherein, the combined partition includes one color partition corresponding to each preset color channel; the color partition includes a preset number of consecutive preset color values ​​under each preset color channel; the preset color values ​​in each color partition corresponding to the preset color channel are not repeated.

[0143] Step 2: Based on the first color information, determine the first combined partition corresponding to the first pixel from each combined partition; wherein, the first combined partition is a combined partition in which the color partition under each preset color channel is consistent with the first color partition corresponding to the first pixel under the preset color channel; the first color partition is the color partition in which the preset color value is consistent with the color value under the preset color channel in the first color information is located.

[0144] Step 3: Based on the second color information, determine the second combined partition corresponding to the second pixel from each combined partition; wherein, the second combined partition is a combined partition in which the color partition under each preset color channel is consistent with the second color partition corresponding to the second pixel under the preset color channel; the second color partition is the color partition in which the preset color value is consistent with the color value under the preset color channel in the second color information is located.

[0145] Step 4: Concatenate the first number corresponding to each combined partition to obtain the image vector corresponding to the UI design image; and for each UI snapshot, concatenate the second number corresponding to each combined partition to obtain the snapshot vector corresponding to the UI snapshot; where the first number is the total number of all first pixels corresponding to the combined partition; and the second number is the total number of all second pixels corresponding to each combined partition in each UI snapshot.

[0146] Specifically, based on the server, for each preset color channel, such as red (R), green (G), and blue (B), all preset color values ​​under each preset color channel are first partitioned to obtain multiple color partitions corresponding to each preset color channel. Each color partition includes a preset number of consecutive preset color values ​​under each preset color channel, and the preset color values ​​in each color partition corresponding to each preset color channel are mutually exclusive. The color partitions corresponding to each preset color channel are then combined to obtain multiple combined partitions, each of which includes one color partition corresponding to each preset color channel. Based on the color information of each pixel in the first three-channel histogram corresponding to the UI design image, i.e., the first color information, for each preset color channel, the color partition containing the preset color value that matches the color value in each preset color channel of the first color information is determined. This is the color partition corresponding to each pixel in the first three-channel histogram under each preset color channel, i.e., the first color partition. The combined partitions that match both the color partitions under each preset color channel and the first color partitions under each preset color channel are determined as the combined partitions corresponding to each pixel in the first three-channel histogram, i.e., the first combined partitions. Simultaneously, for the color information of each pixel in the second and third channel histogram corresponding to each UI snapshot, i.e., the second color information, firstly, for each preset color channel, determine the color partition where the preset color value that matches the color value in each preset color channel of the second color information is located. This is the color partition corresponding to each pixel in the second and third channel histogram under each preset color channel, i.e., the second color partition. Then, determine the combination partition where both the color partition under each preset color channel and the corresponding second color partition under each preset color channel are consistent, i.e., the combination partition corresponding to each pixel in the second and third channel histogram, i.e., the second combination partition. For all pixels in the UI design image, determine the total number of pixels in the UI design image corresponding to each combination partition, which is the first number of each combination partition. Concatenate the first numbers corresponding to each combination partition to obtain the image vector corresponding to the UI design image. Simultaneously, for all pixels in each UI snapshot, determine the total number of pixels in that UI snapshot corresponding to each combination partition, which is the second number of each combination partition corresponding to each UI snapshot. Concatenate the second numbers of each combination partition corresponding to each UI snapshot to obtain the snapshot vector corresponding to each UI snapshot.

[0147] For example, Figure 6 A flowchart illustrating a user interface testing scheme provided in an embodiment of this application is shown below. Figure 6As shown, for each channel of red (R), green (G), and blue (B), each channel has 256 values. Each RGB channel can be divided into four color partitions: 0-63 (partition 0), 64-127 (partition 1), 128-191 (partition 2), and 192-255 (partition 3). Combining each partition of each RGB channel yields 64 possible combinations. Each color belongs to one of these 64 combinations. In the UI design image, pixel 1 has color information of (62, 254, 172), and pixel 2 has color information of (30, 30, 30). Therefore, pixel 1 belongs to the combination partition (0, 3, 2), and pixel 2 belongs to the combination partition (3, 3, 2). The partition is (0, 0, 0), and so on, until the last pixel n (255, 255, 255) is located in the combined partition (3, 3, 3). For this UI design image, the number of pixels in each combined partition is as follows: for example, the number of pixels in combined partitions (0, 0, 0), (0, 0, 1), (0, 0, 2), ..., (3, 3, 3) are 3, 7, 4, ..., 5 respectively. Then, the pixel counts of each combined partition are concatenated to obtain a 64-dimensional vector (3, 7, 4, ..., 5) for the UI design image. Furthermore, the snapshot vector for each UI snapshot is calculated using this method. The number of color partitions can be adjusted, but the more color partitions there are, the more complex the calculation becomes.

[0148] 407. The server determines the cosine similarity between the image vector and the snapshot vector.

[0149] For example, the server calculates the cosine similarity between the image vector and each snapshot vector based on a cosine similarity algorithm; for example, it can be calculated using a formula. Calculate the cosine similarity cosθ between the image vector and each snapshot vector; where A i Let B be the vector value of the i-th dimension in the image vector. i Let be the vector value of the i-th dimension in the snapshot vector.

[0150] 408. The server determines the UI snapshot corresponding to the snapshot vector with the cosine similarity value closest to 1, which is the similar UI snapshot.

[0151] For example, the server determines the cosine similarity with the closest value to 1 from the cosine similarity between the calculated image vector and each snapshot vector, and takes the UI snapshot corresponding to the snapshot vector with the cosine similarity as the similar UI snapshot corresponding to the image vector.

[0152] 409. The server determines the first element feature in the image features of the UI design image; and determines the second element feature in the image features of the similar UI snapshot; wherein, the first element feature is the element feature of each page element in the UI design image; and the second element feature is the element feature of each page element in the similar UI snapshot.

[0153] For example, the server extracts image features from a UI design image based on an image feature extraction algorithm, and then extracts element features of each page element in the UI design image, i.e., the first element features; simultaneously, based on the same image feature extraction algorithm, it extracts image features resembling a UI snapshot, and then extracts element features of each page element in the UI snapshot, i.e., the second element features. These element features may include, but are not limited to, descriptive data such as the position, size, and color of the relevant elements.

[0154] In one example, step 409 includes:

[0155] The first step is to perform image segmentation processing on the UI design image and the similar UI snapshot respectively, to obtain at least one first graphical user interface (GUI) element image corresponding to the UI design image and at least one second GUI element image corresponding to the similar UI snapshot.

[0156] The second step involves performing feature processing on each first GUI element image to obtain first element features; and performing feature processing on each second GUI element image to obtain second element features.

[0157] For example, the server, based on an object detection algorithm, identifies at least one first graphical user interface (GUI) element corresponding to the UI design image and performs image segmentation processing to obtain at least one first GUI element image corresponding to the UI design image. Simultaneously, based on the same object detection algorithm, it identifies at least one second GUI element corresponding to a similar UI snapshot and performs image segmentation processing to obtain at least one second GUI element image corresponding to the similar UI snapshot. Using image processing techniques, feature processing is performed on all first GUI element images to obtain first element features corresponding to the UI design image; similarly, feature processing is performed on all second GUI element images to obtain second element features corresponding to the similar UI snapshot.

[0158] In one example, the first step of step 409 includes:

[0159] Step 1: Perform edge detection on the UI design image and similar UI snapshots to obtain the original contour information of the UI design image and the snapshot contour information of the similar UI snapshots. The original contour information includes at least one design contour image, which represents the background contour of the UI elements in the UI design image. The snapshot contour information includes at least one snapshot contour image, which represents the background contour of the UI elements in the similar UI snapshots.

[0160] Step 2: Cut the design outline image to obtain the first GUI element image; and cut the snapshot outline image to obtain the second GUI element image.

[0161] Specifically, the server uses an edge detection algorithm to identify each UI element in the UI design image and performs edge detection processing to obtain the background outline of each UI element, i.e., the design outline image of each UI element. The design outline images of all UI elements in the UI design image constitute the original outline information. Simultaneously, based on the same edge detection algorithm, the server identifies each UI element in similar UI snapshots and performs edge detection processing to obtain the background outline of each UI element in the similar UI snapshots, i.e., the snapshot outline image of each UI element. The snapshot outline images of all UI elements in the similar UI snapshots constitute the snapshot outline information. Using image segmentation technology, each design outline image of the UI design image is horizontally and vertically segmented to obtain an image of each first GUI element corresponding to the UI design image. Simultaneously, each snapshot outline image of the similar UI snapshots is horizontally and vertically segmented to obtain an image of each second GUI element corresponding to the similar UI snapshots.

[0162] For example, in a web page, content is displayed in a hierarchical manner, divided into background and foreground. The first step is to find connected regions in the background to detect the edges of elements. To prevent content from being displayed incorrectly, the background is often a solid color or gradient, relatively uniform, and with clearly defined edges. Therefore, the Laplacian operator is first used to perform edge detection on the design draft and UI snapshot to roughly extract the background outlines of each element. The UI page is basically a vertical flow layout, so the results obtained in the previous step can be used for horizontal and vertical slicing to obtain images of each GUI element in the design draft and each GUI element in the UI snapshot, such as input box images, dropdown box images, etc. The coordinates of the cutting points in the UI design image during the slicing process, as well as the coordinates of the cutting points in the similar UI snapshots during the slicing process, are recorded for further processing.

[0163] In one example, the first step of step 409 further includes: performing a flood filling process on the design outline image and the snapshot outline image to obtain the processed design outline image and the processed snapshot outline image.

[0164] For example, edge detection results often contain a lot of noise, requiring further processing to refine the details. A flood fill algorithm is used, with appropriate seed nodes and threshold ranges set, to perform flood fill processing on each design contour image and each snapshot contour image separately, resulting in processed design contour images and processed snapshot contour images. This further connects the background regions and reduces noise.

[0165] In one example, the second step of step 409 includes:

[0166] Step 1: Perform target detection processing on the first GUI element image and the second GUI element image respectively to obtain first component data and second component data; wherein, the first component data represents the foreground component in the first GUI element image; and the second component data represents the foreground component in the second GUI element image.

[0167] Step 2: Extract attributes from the foreground components in each first GUI element image to obtain component attribute features in the first element features; and extract attributes from the foreground components in each second GUI element image to obtain component attribute features in the second element features.

[0168] Step 3: Determine the element position features in the first element features based on the coordinates of the cutting points of each first GUI element image; and determine the element position features in the second element features based on the coordinates of the cutting points of each second GUI element image.

[0169] Specifically, the server performs object detection processing on each first GUI element image in the UI design image based on an object detection processing algorithm, detecting the foreground components, i.e., the first component data, in each first GUI element image of the UI design image. Simultaneously, it performs object detection processing on each second GUI element image in the similar UI snapshot, detecting the foreground components in each second GUI element image of the similar UI snapshot. Through a feature extraction algorithm, attribute feature extraction processing is performed on the foreground components in each first GUI element image of the UI design image, obtaining the component attribute features in the first element features corresponding to the UI design image. Simultaneously, attribute feature extraction processing is performed on the foreground components in each second GUI element image of the similar UI snapshot, obtaining the component attribute features in the second element features corresponding to the similar UI snapshot. By performing horizontal and vertical slicing on each design outline image of the UI design image and each snapshot outline image of a similar UI snapshot based on image slicing technology, the cutting point of each design outline image of the UI design image has the cutting point coordinates in the UI design image, and the cutting point of each snapshot outline image of the similar UI snapshot has the cutting point coordinates in the similar UI snapshot image. Therefore, the element position feature in the first element feature corresponding to the UI design image can be obtained based on the cutting point coordinates of each first GUI element image in the UI design image; at the same time, the element position feature in the second element feature corresponding to the similar UI snapshot can be obtained based on the cutting point coordinates of each second GUI element image in the similar UI snapshot.

[0170] For example, when using object detection algorithms to identify components in the foreground, different companies have slight differences in page and component design styles. To enhance the accuracy of object detection, pages and components from internal systems are selected as sample data. The Faster R-CNN algorithm framework is trained to identify foreground components in GUI element images. The identified components are then attribute-extracted to generate Domain-Specific Language (DSL) JSON data. The extracted content is shown in Table 1 below.

[0171] Table 1

[0172] width width Font size fontSize high height Font fontFamily Rounded corners radius Font color fontColor Border width borderWidth Font alignment textAlign Border type borderType Font spacing letterSpace Border color borderColor Xinggao lineHeight shadow boxShadow Inner spacing padding Background color background

[0173] 410. The server generates UI test results for the user device based on the first element feature and the second element feature.

[0174] For example, the server performs feature comparison processing on the first element features of the UI design image and the second element features of similar UI snapshots based on the feature comparison algorithm, compares the differences in style, attributes and other content of each element, and finally obtains the result of the UI reproduction test.

[0175] In one example, step 410 includes:

[0176] The first step is to generate a first feature tree corresponding to the UI design image based on the first element features; and to generate a second feature tree corresponding to the similar UI snapshot based on the second element features. The first feature tree represents the element hierarchy of each UI element in the UI design image and the inclusion relationship between each UI element; the second feature tree represents the hierarchy, attributes, and inclusion relationship between each UI element in the similar UI snapshot.

[0177] The second step involves traversing and comparing the first and second feature trees to obtain the UI test results for the user device.

[0178] Specifically, the server uses the DSL TREE generation algorithm to parse and process the first element features of the UI design image, generating a DSL TREE (first feature tree) corresponding to the UI design image to represent the element hierarchy of each UI element and the inclusion relationships between them. Simultaneously, based on the same DSL TREE generation algorithm, the server parses and processes the second element features of similar UI snapshots, generating a DSL TREE (second feature tree) corresponding to those snapshots to represent the element hierarchy of each UI element and the inclusion relationships between them. The server traverses the obtained first and second feature trees, compares the differences in the element features of each element, and thus obtains the UI fidelity test results for the user device.

[0179] For example, Figure 7 A schematic diagram illustrating a DSL JSON example provided in this application embodiment, as shown below. Figure 7 As shown, based on the preceding analysis steps, the DSL JSON layout description of the UI design image and similar UI snapshots is obtained, that is, the element characteristics of each element. According to the element hierarchy and inclusion relationship described in the DSL JSON, the corresponding DSL TREE is generated. Figure 8 This is a schematic diagram illustrating a UI fidelity test scenario provided in an embodiment of this application, as shown below. Figure 8 As shown, the two DSL TREEs corresponding to the UI design image and the similar UI snapshot are traversed, and the differences in style, attributes and other content of each element are compared to obtain the UI fidelity test results.

[0180] In this embodiment, based on the above embodiments, on the one hand, in the real operating environment of the WEB application, the snapshot of the page is captured by canvas technology, and the obtained fidelity test results can include the impact of the device model; on the other hand, image recognition technology is used to segment page elements, extract element features to generate feature trees, and the fidelity test results are automatically completed by traversing the feature trees. Thus, UI fidelity testing can be automatically tested without writing test scripts, reducing the threshold for users.

[0181] Figure 9 A schematic diagram of the structure of an image-based user interface testing device provided in this application embodiment is shown below. Figure 9 As shown, the device is used in a server; the device includes:

[0182] The acquisition module 501 is used to receive snapshot timing data sent by the user equipment and acquire the design user interface (UI) image of the user equipment; wherein, the UI design image is the original design UI image of the user equipment; the snapshot timing data includes at least one UI snapshot of the user equipment at a given moment;

[0183] The determination module 502 is used to determine the UI snapshot with the highest similarity to the UI design image from the snapshot time series data, and to identify the similar UI snapshots to the UI design image;

[0184] The generation module 503 is used to generate UI test results for the user device based on the image features of the UI design image and the image features of similar UI snapshots; wherein, the UI test results are the UI fidelity test results for the user device.

[0185] In one possible implementation, the generation module 503 is specifically configured to: determine a first element feature in the image features of the UI design image; and determine a second element feature in the image features of the similar UI snapshot; wherein the first element feature is the element feature of each page element in the UI design image; the second element feature is the element feature of each page element in the similar UI snapshot; and generate UI test results for the user device based on the first element feature and the second element feature.

[0186] In one possible implementation, the generation module 503 is specifically used to: perform image segmentation processing on the UI design image and the similar UI snapshot respectively to obtain at least one first graphical user interface (GUI) element image corresponding to the UI design image and at least one second GUI element image corresponding to the similar UI snapshot; perform feature processing on each first GUI element image to obtain first element features; and perform feature processing on each second GUI element image to obtain second element features.

[0187] In one possible implementation, the generation module 503 is specifically configured to: perform edge detection on the UI design image and the similar UI snapshot respectively to obtain the original contour information of the UI design image and the snapshot contour information of the similar UI snapshot; wherein, the original contour information includes at least one design contour image; the design contour image represents the background contour of the UI element in the UI design image; the snapshot contour information includes at least one snapshot contour image; the snapshot contour image represents the background contour of the UI element in the similar UI snapshot; perform segmentation processing on the design contour image to obtain a first GUI element image; and perform segmentation processing on the snapshot contour image to obtain a second GUI element image.

[0188] In one possible implementation, the generation module 503 is further specifically used to: perform a flood filling process on the design outline image and the snapshot outline image to obtain the processed design outline image and the processed snapshot outline image.

[0189] In one possible implementation, the generation module 503 is further specifically configured to: perform target detection processing on the first GUI element image and the second GUI element image respectively to obtain first component data and second component data; wherein, the first component data represents the foreground component in the first GUI element image; the second component data represents the foreground component in the second GUI element image; perform attribute extraction processing on the foreground component data in each first GUI element image to obtain component attribute features in the first element features; and perform attribute extraction processing on the foreground component data in each second GUI element image to obtain component attribute features in the second element features; determine the element position features in the first element features based on the cutting point coordinates of each first GUI element image; and determine the element position features in the second element features based on the cutting point coordinates of each second GUI element image.

[0190] In one possible implementation, the generation module 503 is further specifically configured to: generate a first feature tree corresponding to the UI design image based on the first element features; and generate a second feature tree corresponding to the similar UI snapshot based on the second element features; wherein the first feature tree represents the element hierarchy of each UI element in the UI design image and the inclusion relationship between each UI element; the second feature tree represents the hierarchy, attributes, and inclusion relationship between each UI element in the similar UI snapshot; and perform traversal comparison processing on the first feature tree and the second feature tree to obtain the UI test result of the user device.

[0191] In one possible implementation, the determining module 502 is specifically used to: determine the image vector corresponding to the UI design image; and determine the snapshot vector set corresponding to the snapshot time series data; wherein the snapshot vector set includes the snapshot vector corresponding to each UI snapshot; determine the cosine similarity between the image vector and the snapshot vector; and determine the UI snapshot corresponding to the snapshot vector whose cosine similarity value is closest to 1, which is the similar UI snapshot.

[0192] In one possible implementation, the determining module 502 is specifically configured to: determine a first three-channel histogram corresponding to the UI design image, and determine a second three-channel histogram corresponding to the UI snapshot; wherein, the first three-channel histogram includes first color information of each first pixel in the UI design image, the first color information including the color value of the first pixel in each preset color channel; the second three-channel histogram includes second color information of each second pixel in the UI snapshot, the second color information including the color value of the second pixel in each preset color channel; determine the image vector corresponding to the UI design image based on the first three-channel histogram; and determine the snapshot vector corresponding to the UI snapshot based on the second three-channel histogram.

[0193] In one possible implementation, the determining module 502 is specifically configured to: perform partitioning and combination processing on at least one preset color value corresponding to each preset color channel to obtain at least one combined partition; wherein, the combined partition includes a color partition corresponding to each preset color channel; the color partition includes a preset number of consecutive preset color values ​​under each preset color channel; the preset color values ​​in each color partition corresponding to the preset color channel are not repeated; and determine the first combined partition corresponding to the first pixel from each combined partition according to the first color information; wherein, the first combined partition is a combined partition in which the color partition under each preset color channel and the first color partition corresponding to the first pixel under the preset color channel are both consistent; the first color partition is a combination partition in which the preset color values ​​are consistent with the color values ​​under the preset color channel in the first color information. The color partition is defined; based on the second color information, the second combined partition corresponding to the second pixel is determined from each combined partition; wherein, the second combined partition is a combined partition in which the color partition under each preset color channel is consistent with the second color partition corresponding to the second pixel under the preset color channel; the second color partition is the color partition in which the preset color value is consistent with the color value under the preset color channel in the second color information is located; the first number corresponding to each combined partition is concatenated to obtain the image vector corresponding to the UI design image; and for each UI snapshot, the second number corresponding to each combined partition is concatenated to obtain the snapshot vector corresponding to the UI snapshot; wherein, the first number is the total number of all first pixels corresponding to the combined partition; the second number is the total number of all second pixels corresponding to each combined partition in each UI snapshot.

[0194] The apparatus in this embodiment can execute the technical solutions in the above method. Its specific implementation process and technical principles are the same, and will not be repeated here.

[0195] Figure 10 A schematic diagram of another image-based user interface testing device provided in this application embodiment is shown below. Figure 10 As shown, the device is applied to a user equipment; the device includes:

[0196] The drawing module 601 is used to create initial drawing elements and draw each page element in the user interface (UI) of the user device at the current moment into the initial drawing elements to obtain the drawing elements;

[0197] The generation module 602 is used to export data from the drawing elements, generate a UI snapshot of the user device at the current moment, and add the UI snapshot to the snapshot time series data.

[0198] The sending module 603 is used to send snapshot time series data to a server; wherein the server includes the means as described in the third aspect.

[0199] In one possible implementation, the drawing module 601 is specifically used to: parse and process the page currently displayed by the user device through a preset Document Object Model (DOM) tree to obtain an element set; wherein the element set includes the style and layout information of each page element in the user interface (UI) of the user device; and draw each page element in the element set onto the initial drawing element according to the DOM tree and the element set to obtain the drawing element.

[0200] In one possible implementation, the apparatus is further configured to: if it is determined that the number of snapshots in the snapshot time series data is greater than or equal to a preset number, then stop performing document object parsing processing on the user interface (UI) of the user device; wherein the number of snapshots is the total number of UI snapshots in the snapshot time series data.

[0201] The apparatus in this embodiment can execute the technical solutions in the above method. Its specific implementation process and technical principles are the same, and will not be repeated here.

[0202] Figure 11 This application provides a schematic diagram of the structure of a server, as shown in the embodiment of the present application. Figure 11 As shown, the server includes: a memory 701 and a processor 702; the memory 701 is used to store instructions executable by the processor 702.

[0203] The processor 702 is configured to perform the method provided in the above embodiments.

[0204] The server also includes a receiver 703 and a transmitter 704. The receiver 703 is used to receive instructions and data sent by other devices, and the transmitter 704 is used to send instructions and data to external devices.

[0205] Figure 12 This is a schematic diagram of the structure of a user equipment provided in an embodiment of this application, such as... Figure 12 As shown, the user equipment includes: a memory 801 and a processor 802; the memory 801 is used to store instructions executable by the processor 802.

[0206] The processor 802 is configured to perform the methods provided in the above embodiments.

[0207] The user equipment also includes a receiver 803 and a transmitter 804. The receiver 803 is used to receive instructions and data sent by other devices, and the transmitter 804 is used to send instructions and data to external devices.

[0208] The specific implementation process of the processor can be found in the above method embodiments, and its implementation principle and technical effect are similar, so it will not be repeated here.

[0209] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0210] This application also provides a chip for executing instructions, which is used to execute the technical solutions of the processing methods described above.

[0211] This application also provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are run on a computer, the computer executes the technical solution of the processing method described in the above embodiments.

[0212] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0213] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0214] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium, and when the at least one processor executes the computer program, it can implement the technical solution of the processing method in the above embodiments.

[0215] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0216] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0217] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0218] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0219] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. An image-based user interface testing method, characterized in that, The method is applied to a server; the method includes: The system receives snapshot time-series data sent by the user equipment and acquires the user interface (UI) design image of the user equipment; wherein the UI design image is the original design of the user equipment's UI image; the snapshot time-series data includes at least one UI snapshot of the user equipment at any given moment, and each UI snapshot is generated by drawing each page element in the UI at the current moment onto an initially created drawing element and performing data export processing on the drawing element; The UI snapshot with the highest similarity to the UI design image is determined from the snapshot time series data and is taken as the similar UI snapshot to the UI design image; The UI design image and the similar UI snapshot are respectively subjected to image segmentation processing to obtain at least one first graphical user interface (GUI) element image corresponding to the UI design image and at least one second GUI element image corresponding to the similar UI snapshot; The first GUI element image is processed to obtain a first element feature; and the second GUI element image is processed to obtain a second element feature; wherein, the first element feature is the element feature of each page element in the UI design image; and the second element feature is the element feature of each page element in a similar UI snapshot; Based on the first element feature and the second element feature, the UI test result of the user device is generated; wherein, the UI test result is the UI fidelity test result of the user device; Specifically, determining the UI snapshot with the highest similarity to the UI design image from the snapshot time-series data, as the similar UI snapshot to the UI design image, includes: Determine the image vector corresponding to the UI design image; and determine the snapshot vector set corresponding to the snapshot time series data; wherein, the snapshot vector set includes the snapshot vector corresponding to each UI snapshot; Determine the cosine similarity between the image vector and the snapshot vector; The UI snapshot corresponding to the snapshot vector whose cosine similarity value is closest to 1 is determined as the similar UI snapshot.

2. The method according to claim 1, characterized in that, Image segmentation processing is performed on the UI design image and the similar UI snapshot to obtain at least one first graphical user interface (GUI) element image corresponding to the UI design image and at least one second GUI element image corresponding to the similar UI snapshot, including: Edge detection is performed on the UI design image and the similar UI snapshot respectively to obtain the original contour information of the UI design image and the snapshot contour information of the similar UI snapshot; wherein, the original contour information includes at least one design contour image; the design contour image represents the background contour of the UI element in the UI design image; the snapshot contour information includes at least one snapshot contour image; the snapshot contour image represents the background contour of the UI element in the similar UI snapshot; The design outline image is segmented to obtain the first GUI element image; and the snapshot outline image is segmented to obtain the second GUI element image.

3. The method according to claim 2, characterized in that, The method further includes: The design outline image and the snapshot outline image are subjected to a flood filling process to obtain the processed design outline image and the processed snapshot outline image.

4. The method according to claim 1, characterized in that, Perform feature processing on each of the first GUI element images to obtain the features of the first element; The second GUI element images are then subjected to feature processing to obtain the features of the second elements, including: The first GUI element image and the second GUI element image are subjected to target detection processing respectively to obtain first component data and second component data; wherein, the first component data represents the foreground component in the first GUI element image; and the second component data represents the foreground component in the second GUI element image. Attribute extraction processing is performed on the foreground component data in each of the first GUI element images to obtain the component attribute features in the first element features; and attribute extraction processing is performed on the foreground component data in each of the second GUI element images to obtain the component attribute features in the second element features; Based on the coordinates of the cutting points of each of the first GUI element images, the element position features in the first element features are determined; and based on the coordinates of the cutting points of each of the second GUI element images, the element position features in the second element features are determined.

5. The method according to claim 1, characterized in that, Based on the first element feature and the second element feature, the UI test results of the user device are generated, including: Based on the first element features, a first feature tree corresponding to the UI design image is generated; and based on the second element features, a second feature tree corresponding to the similar UI snapshot is generated; wherein, the first feature tree represents the element hierarchy of each UI element in the UI design image and the inclusion relationship between each UI element; the second feature tree represents the hierarchy, attributes, and inclusion relationship between each UI element in the similar UI snapshot; The first feature tree and the second feature tree are traversed and compared to obtain the UI test results of the user device.

6. The method according to claim 1, characterized in that, Determine the image vector corresponding to the UI design image; and determine the set of snapshot vectors corresponding to the snapshot time series data, including: A first three-channel histogram corresponding to the UI design image is determined, and a second three-channel histogram corresponding to the UI snapshot is determined; wherein, the first three-channel histogram includes first color information of each first pixel in the UI design image, and the first color information includes the color value of the first pixel in each preset color channel; the second three-channel histogram includes second color information of each second pixel in the UI snapshot, and the second color information includes the color value of the second pixel in each preset color channel; Based on the first three-channel histogram, the image vector corresponding to the UI design image is determined; and based on the second three-channel histogram, the snapshot vector corresponding to the UI snapshot is determined.

7. The method according to claim 6, characterized in that, Based on the first three-channel histogram, determine the image vector corresponding to the UI design image; and based on the second three-channel histogram, determine the snapshot vector corresponding to the UI snapshot, including: At least one preset color value corresponding to each preset color channel is partitioned and combined to obtain at least one combined partition; wherein, the combined partition includes a color partition corresponding to each preset color channel; the color partition includes a preset number of consecutive preset color values ​​under each preset color channel; the preset color values ​​in each color partition corresponding to the preset color channel are not repeated. Based on the first color information, a first combined partition corresponding to the first pixel is determined from each of the combined partitions; wherein, the first combined partition is a combined partition in which the color partition under each preset color channel is consistent with the first color partition corresponding to the first pixel under the preset color channel; the first color partition is the color partition in which the preset color value is consistent with the color value under the preset color channel in the first color information is located. Based on the second color information, a second combined partition corresponding to the second pixel is determined from each of the combined partitions; wherein, the second combined partition is a combined partition in which the color partition under each preset color channel is consistent with the second color partition corresponding to the second pixel under the preset color channel; the second partition is the color partition in which the preset color value is consistent with the color value under the preset color channel in the second color information is located. The first number corresponding to each of the combined partitions is concatenated to obtain the image vector corresponding to the UI design image; and for each UI snapshot, the second number corresponding to each of the combined partitions is concatenated to obtain the snapshot vector corresponding to the UI snapshot; wherein, the first number is the total number of all first pixels corresponding to the combined partition; and the second number is the total number of all second pixels corresponding to each combined partition in each UI snapshot.

8. A method for testing user interfaces based on images, characterized in that, The method is applied to a user equipment; the method includes: Create an initial drawing element, and draw each page element of the user interface (UI) of the user device at the current moment onto the initial drawing element to obtain the drawing element; The drawing elements are processed to export data, generating a UI snapshot of the user device at the current moment; and the UI snapshot is added to the snapshot time series data. The snapshot time-series data is sent to a server; wherein the server is used to perform the method as described in any one of claims 1-7.

9. The method according to claim 8, characterized in that, Each page element in the user interface (UI) of the user device at the current moment is drawn onto the initial drawing element to obtain the drawing element, including: The page currently displayed on the user device is parsed and processed using a preset Document Object Model (DOM) tree to obtain an element set; wherein, the element set includes the style and layout information of each page element in the user interface (UI) of the user device; Based on the DOM tree and the element set, each page element in the element set is drawn onto the initial drawing element to obtain the drawing element.

10. The method according to claim 8 or 9, characterized in that, The method further includes: If it is determined that the number of snapshots in the snapshot time series data is greater than or equal to a preset number, then the document object parsing process for the user interface (UI) of the user device is stopped; wherein, the number of snapshots is the total number of UI snapshots in the snapshot time series data.

11. An image-based user interface testing device, characterized in that, The device is used in a server; the device includes: The acquisition module is used to receive snapshot time-series data sent by the user device and acquire the user interface (UI) design image of the user device; wherein, the UI design image is the original design of the user device's UI image; the snapshot time-series data includes at least one UI snapshot of the user device at a given moment, and each UI snapshot is generated by drawing each page element in the UI at the current moment onto an initially created drawing element and performing data export processing on the drawing element; The determination module is used to determine the UI snapshot with the highest similarity to the UI design image from the snapshot time series data, and to identify the similar UI snapshot to the UI design image. The generation module performs image segmentation processing on the UI design image and the similar UI snapshot to obtain at least one first graphical user interface (GUI) element image corresponding to the UI design image and at least one second GUI element image corresponding to the similar UI snapshot; performs feature processing on each first GUI element image to obtain first element features; and performs feature processing on each second GUI element image to obtain second element features; wherein, the first element features are the element features of each page element in the UI design image; the second element features are the element features of each page element in the similar UI snapshot; and generates the UI test result of the user device based on the first element features and the second element features; wherein, the UI test result is the UI fidelity test result of the user device; The determining module is specifically used to: determine the image vector corresponding to the UI design image; and determine the snapshot vector set corresponding to the snapshot time series data; wherein, the snapshot vector set includes the snapshot vector corresponding to each UI snapshot; Determine the cosine similarity between the image vector and the snapshot vector; The UI snapshot corresponding to the snapshot vector whose cosine similarity value is closest to 1 is determined as the similar UI snapshot.

12. An image-based user interface testing device, characterized in that, The device is used in a user equipment; the device includes: A drawing module is used to create initial drawing elements and draw each page element in the user interface (UI) of the user device at the current moment onto the initial drawing elements to obtain drawing elements; The generation module is used to export data from the drawing elements, generate a UI snapshot of the user device at the current moment, and add the UI snapshot to the snapshot time series data. A sending module is used to send the snapshot time-series data to a server; wherein the server includes the apparatus as described in claim 11.

13. A server, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.

14. A user equipment, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 8-10.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-10.

16. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-10.

Images