Image-based user interface test method and device, equipment and storage medium

Abstract

The embodiment of the invention provides an image-based user interface test method and device, equipment and a storage medium. The method comprises the following steps: receiving snapshot time sequence data sent by user equipment, and obtaining a user interface (UI) design image of the user equipment; wherein the UI design image is a UI image of user equipment which is originally designed; the snapshot time sequence data comprises UI snapshots of the user equipment at at least one moment; determining a UI snapshot with the highest similarity with the UI design image from the snapshot time sequence data, wherein the UI snapshot is a similar UI snapshot of the UI design image; generating a UI test result of the user equipment according to the image features of the UI design image and the image features of the similar UI snapshots; wherein the UI test result is a UI reduction degree test result of the user equipment. The method is used for achieving the effect of improving the UI test efficiency and accuracy.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to an image-based user interface testing method, device, equipment and storage medium. Background Art

[0002] After the software product is developed, the user interface (UI) needs to be tested for UI restoration to ensure that the actual interface is visually consistent with the original design interface in order to control product quality and user experience.

[0003] In the prior art, testers manually compare the actual interface with the original design interface to find differences in order to complete the UI restoration test.

[0004] However, in the above method, a manual comparison method is adopted, which requires repeated manual comparison operations, resulting in reduced efficiency and accuracy of UI testing. Summary of the invention

[0005] The embodiments of the present application provide an image-based user interface testing method, apparatus, device, and storage medium to achieve the effect of improving the efficiency and accuracy of UI testing.

[0006] In a first aspect, an embodiment of the present application provides an image-based user interface testing method, which is applied to a server; the method comprises:

[0007] Receive snapshot time series data sent by a user device, and obtain a user interface UI design image of the user device; wherein the UI design image is an originally designed UI image of the user device; the snapshot time series data includes a UI snapshot of the user device at at least one moment;

[0008] Determine, from the snapshot time series data, a UI snapshot having the highest similarity to the UI design image as a similar UI snapshot of the UI design image;

[0009] A UI test result of the user device is generated according to the image features of the UI design image and the image features of the similar UI snapshot; wherein the UI test result is a UI restoration degree test result of the user device.

[0010] In a possible implementation, generating a UI test result of the user device according to the image features of the UI design image and the image features of the similar UI snapshot includes:

[0011] Determine a first element feature in the image features of the UI design image; and determine a second element feature in the image features of the similar UI snapshot; wherein the first element feature is an element feature of each page element in the UI design image; and the second element feature is an element feature of each page element in the similar UI snapshot;

[0012] Generate a UI test result of the user equipment according to the first element feature and the second element feature.

[0013] In a possible implementation, determining a first element feature in the image features of the UI design image; and determining a second element feature in the image features of the similar UI snapshot include:

[0014] Performing image segmentation processing on the UI design image and the similar UI snapshot respectively to obtain at least one first graphical user interface GUI element picture corresponding to the UI design image and at least one second GUI element picture corresponding to the similar UI snapshot;

[0015] Perform feature processing on each of the first GUI element images to obtain the first element features; and perform feature processing on each of the second GUI element images to obtain the second element features.

[0016] In a possible implementation, performing image segmentation processing on the UI design image and the similar UI snapshot respectively to obtain at least one first graphical user interface GUI element picture corresponding to the UI design image and at least one second GUI element picture corresponding to the similar UI snapshot includes:

[0017] Performing edge detection on the UI design image and the similar UI snapshot respectively, to obtain original contour information of the UI design image and snapshot contour information of the similar UI snapshot; wherein the original contour information includes at least one design contour image; the design contour image represents the background contour of the UI element in the UI design image; the snapshot contour information includes at least one snapshot contour image; the snapshot contour image represents the background contour of the UI element in the similar UI snapshot;

[0018] The design outline image is cut to obtain the first GUI element image; and the snapshot outline image is cut to obtain the second GUI element image.

[0019] In a possible implementation, the method further includes:

[0020] The design contour image and the snapshot contour image are subjected to a flood filling process to obtain a processed design contour image and a processed snapshot contour image.

[0021] In a possible implementation, performing feature processing on each of the first GUI element images to obtain the first element features; and performing feature processing on each of the second GUI element images to obtain the second element features, includes:

[0022] Performing target detection processing on the first GUI element image and the second GUI element image respectively to obtain first component data and second component data; wherein the first component data represents a foreground component in the first GUI element image; and the second component data represents a foreground component in the second GUI element image;

[0023] Performing attribute extraction processing on the foreground component data in each of the first GUI element images to obtain component attribute features in the first element features; and performing attribute extraction processing on the foreground component data in each of the second GUI element images to obtain component attribute features in the second element features;

[0024] According to the cutting point coordinates of each of the first GUI element images, the element position features in the first element features are determined; and according to the cutting point coordinates of each of the second GUI element images, the element position features in the second element features are determined.

[0025] In a possible implementation, generating a UI test result of the user equipment according to the first element feature and the second element feature includes:

[0026] Generate a first feature tree corresponding to the UI design image according to the first element feature; and generate a second feature tree corresponding to the similar UI snapshot according to the second element feature; wherein the first feature tree represents the element hierarchy of each UI element in the UI design image and the inclusion relationship between each UI element; and the second feature tree represents the hierarchy, attributes and inclusion relationship between each UI element in the similar UI snapshot;

[0027] The first feature tree and the second feature tree are traversed and compared to obtain a UI test result of the user equipment.

[0028] In a possible implementation, determining from the snapshot time series data a UI snapshot having the highest similarity to the UI design image as a similar UI snapshot of the UI design image includes:

[0029] Determine an image vector corresponding to the UI design image; and determine a snapshot vector set corresponding to the snapshot time series data; wherein the snapshot vector set includes a snapshot vector corresponding to each of the UI snapshots;

[0030] determining a cosine similarity between the image vector and the snapshot vector;

[0031] Determine the UI snapshot corresponding to the snapshot vector whose cosine similarity value is closest to 1 as the similar UI snapshot.

[0032] In a possible implementation, determining an image vector corresponding to the UI design image; and determining a set of snapshot vectors corresponding to the snapshot time series data include:

[0033] Determine a first three-channel histogram corresponding to the UI design image, and determine a second three-channel histogram corresponding to the UI snapshot; wherein the first three-channel histogram includes first color information of each first pixel in the UI design image, and the first color information includes a color value of the first pixel in each preset color channel; the second three-channel histogram includes second color information of each second pixel in the UI snapshot, and the second color information includes a color value of the second pixel in each preset color channel;

[0034] An image vector corresponding to the UI design image is determined according to the first three-channel histogram; and a snapshot vector corresponding to the UI snapshot is determined according to the second three-channel histogram.

[0035] In a possible implementation, determining the image vector corresponding to the UI design image according to the first three-channel histogram; and determining the snapshot vector corresponding to the UI snapshot according to the second three-channel histogram, includes:

[0036] Performing a partition combination process on at least one preset color value corresponding to each of the preset color channels to obtain at least one combined partition; wherein the combined partition includes a color partition corresponding to each of the preset color channels; the color partition includes a preset number of continuous preset color values ​​under each of the preset color channels; the preset color values ​​in each color partition corresponding to the preset color channel do not overlap with each other;

[0037] According to the first color information, a first combined partition corresponding to the first pixel is determined from each of the combined partitions; wherein the first combined partition is a combined partition in which the color partition under each of the preset color channels is consistent with the first color partition corresponding to the first pixel under the preset color channel; the first color partition is a color partition in which a preset color value consistent with the color value under the preset color channel in the first color information is located;

[0038] According to the second color information, a second combined partition corresponding to the second pixel is determined from each of the combined partitions; wherein the second combined partition is a combined partition in which the color partition under each of the preset color channels is consistent with the second color partition corresponding to the second pixel under the preset color channel; and the second color partition is a color partition in which a preset color value consistent with the color value under the preset color channel in the second color information is located;

[0039] The first numbers corresponding to each of the combined partitions are spliced ​​together to obtain an image vector corresponding to the UI design image; and for each of the UI snapshots, the second numbers corresponding to each of the combined partitions are spliced ​​together to obtain a snapshot vector corresponding to the UI snapshot; wherein the first number is the total number of all first pixel points corresponding to the combined partition; and the second number is the total number of all second pixel points corresponding to each combined partition in each UI snapshot.

[0040] In a second aspect, an embodiment of the present application provides an image-based user interface testing method, which is applied to a user device; the method includes:

[0041] Creating an initial drawing element, and drawing each page element in the user interface UI of the user device at the current moment into the initial drawing element to obtain a drawing element;

[0042] Performing data export processing on the drawing element to generate a UI snapshot of the user device at the current moment; and adding the UI snapshot to the snapshot time series data;

[0043] The snapshot time series data is sent to a server; wherein the server is used to execute the method as described in the first aspect.

[0044] In a possible implementation, drawing each page element in the user interface UI of the user device at the current moment into the initial drawing element to obtain the drawing element includes:

[0045] The page currently displayed by the user device is parsed through a preset document object model DOM tree to obtain an element set; wherein the element set includes the style and layout information of each page element in the user interface UI of the user device;

[0046] According to the DOM tree and the element set, each page element in the element set is drawn into the initial drawing element to obtain the drawing element.

[0047] In a possible implementation, the method further includes:

[0048] If it is determined that the number of snapshots in the snapshot time series data is greater than or equal to a preset number, the document object parsing process for the user interface UI of the user device is stopped; wherein the number of snapshots is the total number of UI snapshots in the snapshot time series data.

[0049] In a third aspect, an embodiment of the present application provides an image-based user interface testing device, which is applied to a server; the device includes:

[0050] An acquisition module, configured to receive snapshot time series data sent by a user device, and acquire a designed user interface UI image of the user device; wherein the UI design image is an originally designed UI image of the user device; and the snapshot time series data includes a UI snapshot of the user device at at least one moment;

[0051] a determination module, configured to determine, from the snapshot time series data, a UI snapshot having the highest similarity to the UI design image, as a similar UI snapshot of the UI design image;

[0052] A generating module is used to generate a UI test result of the user device according to the image features of the UI design image and the image features of the similar UI snapshot; wherein the UI test result is a UI restoration test result of the user device.

[0053] In a possible implementation, the generation module is specifically used to: determine a first element feature in the image features of the UI design image; and determine a second element feature in the image features of the similar UI snapshot; wherein the first element feature is an element feature of each page element in the UI design image; the second element feature is an element feature of each page element in the similar UI snapshot; and generate a UI test result of the user device based on the first element feature and the second element feature.

[0054] In a possible implementation, the generation module is specifically used to: perform image segmentation processing on the UI design image and the similar UI snapshot respectively to obtain at least one first graphical user interface GUI element image corresponding to the UI design image and at least one second GUI element image corresponding to the similar UI snapshot; perform feature processing on each of the first GUI element images to obtain the first element feature; and perform feature processing on each of the second GUI element images to obtain the second element feature.

[0055] In a possible implementation, the generation module is specifically used to: perform edge detection on the UI design image and the similar UI snapshot respectively to obtain original contour information of the UI design image and snapshot contour information of the similar UI snapshot; wherein the original contour information includes at least one design contour image; the design contour image represents the background contour of the UI element in the UI design image; the snapshot contour information includes at least one snapshot contour image; the snapshot contour image represents the background contour of the UI element in the similar UI snapshot; perform cutting processing on the design contour image to obtain the first GUI element image; and perform cutting processing on the snapshot contour image to obtain the second GUI element image.

[0056] In a possible implementation manner, the generating module is further specifically configured to: perform a flood filling process on the design contour image and the snapshot contour image to obtain a processed design contour image and a processed snapshot contour image.

[0057] In a possible implementation, the generation module is further specifically used to: perform target detection processing on the first GUI element image and the second GUI element image respectively to obtain first component data and second component data; wherein the first component data represents the foreground component in the first GUI element image; the second component data represents the foreground component in the second GUI element image; perform attribute extraction processing on the foreground component data in each of the first GUI element images to obtain component attribute features in the first element features; and perform attribute extraction processing on the foreground component data in each of the second GUI element images to obtain component attribute features in the second element features; determine the element position features in the first element features based on the cutting point coordinates of each of the first GUI element images; and determine the element position features in the second element features based on the cutting point coordinates of each of the second GUI element images.

[0058] In a possible implementation, the generation module is further specifically used to: generate a first feature tree corresponding to the UI design image based on the first element feature; and generate a second feature tree corresponding to the similar UI snapshot based on the second element feature; wherein the first feature tree represents the element hierarchy of each UI element in the UI design image and the inclusion relationship between each UI element; the second feature tree represents the hierarchy, attributes of each UI element and the inclusion relationship between each UI element in the similar UI snapshot; and the first feature tree and the second feature tree are traversed and compared to obtain the UI test result of the user device.

[0059] In a possible implementation, the determination module is specifically used to: determine the image vector corresponding to the UI design image; and determine a set of snapshot vectors corresponding to the snapshot time series data; wherein the set of snapshot vectors includes snapshot vectors corresponding to each of the UI snapshots; determine the cosine similarity between the image vector and the snapshot vector; and determine that the UI snapshot corresponding to the snapshot vector whose cosine similarity value is closest to 1 is the similar UI snapshot.

[0060] In a possible implementation, the determination module is specifically used to: determine a first three-channel histogram corresponding to the UI design image, and determine a second three-channel histogram corresponding to the UI snapshot; wherein the first three-channel histogram includes first color information of each first pixel in the UI design image, and the first color information includes the color value of the first pixel under each preset color channel; the second three-channel histogram includes second color information of each second pixel in the UI snapshot, and the second color information includes the color value of the second pixel under each of the preset color channels; determine the image vector corresponding to the UI design image according to the first three-channel histogram; and determine the snapshot vector corresponding to the UI snapshot according to the second three-channel histogram.

[0061] In a possible implementation, the determination module is specifically used to: perform partition combination processing on at least one preset color value corresponding to each of the preset color channels to obtain at least one combined partition; wherein the combined partition includes a color partition corresponding to each of the preset color channels; the color partition includes a preset number of continuous preset color values ​​under each of the preset color channels; the preset color values ​​in each color partition corresponding to the preset color channel do not overlap with each other; according to the first color information, determine the first combined partition corresponding to the first pixel point from each of the combined partitions; wherein the first combined partition is a combined partition in which the color partition under each of the preset color channels is consistent with the first color partition corresponding to the first pixel point under the preset color channel; the first color partition is a combined partition in which the preset color value consistent with the color value under the preset color channel in the first color information is located color partition; according to the second color information, determine the second combined partition corresponding to the second pixel point from each of the combined partitions; wherein the second combined partition is a combined partition whose color partition under each of the preset color channels is consistent with the second color partition corresponding to the second pixel point under the preset color channel; the second color partition is a color partition where a preset color value is located that is consistent with the color value under the preset color channel in the second color information; concatenate the first numbers corresponding to each of the combined partitions to obtain an image vector corresponding to the UI design image; and for each of the UI snapshots, concatenate the second numbers corresponding to each of the combined partitions to obtain a snapshot vector corresponding to the UI snapshot; wherein the first number is the total number of all first pixel points corresponding to the combined partition; and the second number is the total number of all second pixel points corresponding to each combined partition in each UI snapshot.

[0062] In a fourth aspect, an embodiment of the present application provides an image-based user interface testing device, which is applied to a user equipment; the device includes:

[0063] A drawing module, used to create an initial drawing element, and draw each page element in the user interface UI of the user equipment at the current moment into the initial drawing element to obtain a drawing element;

[0064] A generating module, configured to perform data export processing on the drawing element, generate a UI snapshot of the user device at the current moment; and add the UI snapshot to the snapshot time series data;

[0065] A sending module is used to send the snapshot time series data to a server; wherein the server includes the device as described in the third aspect.

[0066] In a possible implementation, the drawing module is specifically used to: parse the page currently displayed by the user device through a preset document object model DOM tree to obtain an element set; wherein the element set includes the style and layout information of each page element in the user interface UI of the user device; and draw each page element in the element set into the initial drawing element according to the DOM tree and the element set to obtain the drawing element.

[0067] In a possible implementation, the device is also used to: if it is determined that the number of snapshots in the snapshot time series data is greater than or equal to a preset number, stop performing document object parsing processing on the user interface UI of the user device; wherein the number of snapshots is the total number of UI snapshots in the snapshot time series data.

[0068] In a fifth aspect, an embodiment of the present application provides a server, including: a memory, a processor;

[0069] The memory stores computer-executable instructions;

[0070] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0071] In a sixth aspect, an embodiment of the present application provides a user equipment, including: a memory, a processor;

[0072] The memory stores computer-executable instructions;

[0073] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above second aspect and / or various possible implementations of the second aspect.

[0074] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-readable storage medium is stored computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method of the first aspect and / or the second aspect as described above.

[0075] In an eighth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements the method of the first aspect and / or the second aspect above when executed by a processor.

[0076] The image-based user interface testing method provided by the embodiment of the present application receives snapshot timing data sent by a user device, and obtains a designed user interface UI image of the user device; wherein the UI design image is a UI image of the user device of the original design; the snapshot timing data includes a UI snapshot of the user device at at least one moment; the UI snapshot with the highest similarity to the UI design image is determined from the snapshot timing data, which is a similar UI snapshot of the UI design image; the UI test result of the user device is generated according to the image features of the UI design image and the image features of the similar UI snapshot; wherein the UI test result is a UI restoration test result of the user device. By obtaining multiple UI snapshots from the user device, and selecting a UI snapshot that is most similar to the UI image of the user device of the original design, and comparing the image features of the most similar UI snapshot with the image features of the UI design image, the result of the automatic test of the UI restoration is obtained, thereby achieving the effect of improving the efficiency and accuracy of the UI test. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0078] Figure 1 A schematic diagram of an application scenario provided for this application;

[0079] Figure 2 A flowchart of an image-based user interface testing method provided in an embodiment of the present application;

[0080] Figure 3 A flowchart of a user interface testing solution provided in an embodiment of the present application;

[0081] Figure 4 A flowchart of another image-based user interface testing method provided in an embodiment of the present application;

[0082] Figure 5 A flowchart of another image-based user interface testing method provided in an embodiment of the present application;

[0083] Figure 6 A flowchart of a user interface testing solution provided in an embodiment of the present application;

[0084] Figure 7 A schematic diagram of a DSL JSON example provided in an embodiment of the present application;

[0085] Figure 8 A schematic diagram of a UI restoration test situation provided in an embodiment of the present application;

[0086] Fig. 9A schematic diagram of the structure of an image-based user interface testing device provided in an embodiment of the present application;

[0087] Fig.10 A schematic diagram of the structure of another image-based user interface testing device provided in an embodiment of the present application;

[0088] Fig.11 A schematic diagram of the structure of a server provided in an embodiment of the present application;

[0089] Fig.12 A schematic diagram of the structure of a user device provided in an embodiment of the present application.

[0090] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0091] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0092] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0093] In addition, this application involves big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.), and the use of artificial intelligence technology for automated decision-making, and a technical solution for making decisions that have a significant impact on personal rights and interests based on the results of automated decision-making. The application provides users with corresponding operation entrances for them to choose to agree or reject the results of automated decision-making; if the user chooses to reject, the expert decision-making process will be entered.

[0094] It should be noted that the present application can be used in the field of artificial intelligence, and can also be used in any field other than artificial intelligence, and the application field of the present application is not limited.

[0095] Figure 1 A schematic diagram of an application scenario provided for this application, such as Figure 1 As shown, the specific application scenario of this application is: after the software product development is completed, the user interface (User Interface, referred to as UI) needs to be tested for UI restoration to ensure that the actual UI interface is visually consistent with the original designed UI interface, so as to control product quality and user experience.

[0096] In combination with the above scenario, it can be seen that in the prior art, testers manually compare the actual interface with the original design interface to find differences to complete the UI restoration test. However, there is a technical problem that the manual comparison method requires repeated manual comparison operations, which reduces the efficiency and accuracy of UI testing.

[0097] The image-based user interface testing method provided by the present application obtains multiple UI snapshots from a user device, selects a UI snapshot that is most similar to the UI image of the original designed user device, compares the image features of the most similar UI snapshot with the image features of the UI design image, and obtains the result of automatic UI restoration test, thereby solving the technical problem of reduced efficiency and accuracy of UI testing.

[0098] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0099] Figure 2 A flowchart of an image-based user interface testing method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the method includes:

[0100] 201. Receive snapshot time series data sent by a user device, and obtain a designed user interface UI image of the user device; wherein the UI design image is an originally designed UI image of the user device; and the snapshot time series data includes a UI snapshot of the user device at at least one moment.

[0101] Exemplarily, the execution subject of this embodiment may be a server. Figure 3 A flowchart of a user interface testing solution provided in an embodiment of the present application is shown in FIG. Figure 3As shown, the user device can generate a plug-in based on the UI snapshot in the real running environment of the WEB application, collect the user interface UI snapshots (C1, C2, ...) of the web page displayed by the user device at one or more moments in the current time period, package them in the snapshot time series data and send them to the server. The server algorithm module based on the server receives the snapshot time series data sent by the user device, and obtains the UI image of the web page displayed by the user device in the original design in the UI design draft, that is, the UI design image, for processing.

[0102] 202. Determine, from the snapshot time series data, a UI snapshot having the highest similarity to the UI design image, as a similar UI snapshot of the UI design image.

[0103] For example, in combination Figure 3 ,Based on the image detection algorithm of the pre-matching module, the server ,screens out the UI snapshots that best match and have the highest ,similarity to the UI design image from the UI snapshot time series data, which ,are similar UI snapshots to the UI design image.

[0104] 203. Generate a UI test result of the user device according to the image features of the UI design image and the image features of the similar UI snapshot; wherein the UI test result is a UI restoration test result of the user device.

[0105] For example, in combination Figure 3 Based on the feature comparison algorithm of the precise comparison module, the server extracts the image features of the UI design image and the image features of similar UI snapshots, such as the key features such as the position, size, color, and rounded corners of each part, and compares the extracted image features of the UI design image with the image features of similar UI snapshots to obtain the UI restoration test results of the user device, that is, the UI test results, such as whether there are obvious differences.

[0106] In this embodiment, an image-based user interface testing method is provided, which obtains multiple UI snapshots from a user device, selects a UI snapshot that is most similar to the UI image of the original designed user device, compares the image features of the most similar UI snapshot with the image features of the UI design image, and obtains the result of automatic UI restoration test, thereby achieving the effect of improving the efficiency and accuracy of UI testing.

[0107] Figure 4 A flowchart of another image-based user interface testing method provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the method includes:

[0108] 301. The user equipment creates an initial drawing element, and draws each page element in the user interface UI of the user equipment at the current moment into the initial drawing element to obtain a drawing element.

[0109] Exemplarily, based on the user device, an initial drawing element, such as a canvas element, is created, and each page element in the user interface UI of the user device at the current moment is obtained, and then all page elements are drawn into the canvas element through canvas technology to obtain the drawn drawing element.

[0110] 302. The user device performs data export processing on the drawing elements to generate a UI snapshot of the user device at the current moment; and adds the UI snapshot to the snapshot time series data.

[0111] Exemplarily, based on the snapshot generation plug-in of the user device, the drawn cavans elements are exported as image data, a UI snapshot of the user device at the current moment is obtained, and the UI snapshot is added to the snapshot time series data.

[0112] 303. The user device sends the snapshot time series data to the server; wherein the server is used to execute the aforementioned image-based user interface testing method.

[0113] Exemplarily, the user device sends all UI snapshots in the generated snapshot time series data to the server, so that the server performs a restoration degree test process on all UI snapshots based on the algorithm module.

[0114] 304. Receive snapshot time series data sent by the user device, and obtain a user interface UI design image of the user device; wherein the UI design image is an originally designed UI image of the user device; and the snapshot time series data includes a UI snapshot of the user device at at least one moment.

[0115] Exemplarily, this step may refer to step 201 and will not be described in detail here.

[0116] 305. The server determines, from the snapshot time series data, a UI snapshot having the highest similarity to the UI design image, as a similar UI snapshot of the UI design image.

[0117] Exemplarily, this step may refer to step 202 and will not be described in detail here.

[0118] 306. The server generates a UI test result of the user device according to the image features of the UI design image and the image features of the similar UI snapshot; wherein the UI test result is a UI restoration test result of the user device.

[0119] Exemplarily, this step may refer to step 203 and will not be described in detail here.

[0120] In this embodiment, based on the above embodiments, on the one hand, the user device completes the snapshot collection of the page with the help of canvas technology in the real running environment of the web application, and the obtained restoration test results can include the impact of the model, thereby improving the accuracy of the UI restoration test results.

[0121] Figure 5 A flowchart of another image-based user interface testing method provided in an embodiment of the present application is shown as follows: Figure 5 As shown, the method includes:

[0122] 401. The user device creates an initial drawing element, and parses the page currently displayed by the user device through a preset document object model DOM tree to obtain an element set; wherein the element set includes the style and layout information of each page element in the user interface UI of the user device.

[0123] Exemplarily, based on the user device, an initial drawing element, such as a canvas element, is created, and a plug-in is generated based on the UI snapshot. Through the preset document object model DOM tree, the DOM tree of the UI of the page of the user device can be parsed every 500ms (the time is customizable), and an element set is calculated, which includes the style and layout information of each element in the UI of the user device, such as hierarchy, size, color, etc.

[0124] 402. The user device draws each page element in the element set into the initial drawing element according to the DOM tree and the element set to obtain a drawing element.

[0125] Exemplarily, based on the canvas technology, the user device draws each page element into the canvas element according to the DOM tree structure and the style information of each page element in the element set, and obtains the drawn cavans element, that is, the drawing element.

[0126] 403. The user device performs data export processing on the drawing elements to generate a UI snapshot of the user device at the current moment; and adds the UI snapshot to the snapshot time series data.

[0127] Exemplarily, this step may refer to step 302 and will not be described in detail here.

[0128] In one example, it also includes: if it is determined that the number of snapshots in the snapshot time series data is greater than or equal to a preset number, stopping document object parsing processing on the user interface UI of the user device; wherein the number of snapshots is the total number of each UI snapshot in the snapshot time series data.

[0129] Exemplarily, when the total number of UI snapshots in the snapshot time series data collected by the page of the user device, that is, the number of snapshots, is greater than or equal to a preset number, the document object parsing process for the user interface UI of the user device is stopped.

[0130] For example, when the page of the user's device collects 10 UI snapshots (the number can be customized), the generation of UI snapshot data for the current page is completed.

[0131] 404. The user device sends the snapshot time series data to the server; wherein the server is used to execute the aforementioned image-based user interface testing method.

[0132] Exemplarily, this step may refer to step 303 and will not be described in detail here.

[0133] 405. The server receives the snapshot timing data sent by the user device, and obtains a user interface UI design image of the user device; wherein the UI design image is an originally designed UI image of the user device; and the snapshot timing data includes a UI snapshot of the user device at at least one moment.

[0134] Exemplarily, this step may refer to step 201 and will not be described in detail here.

[0135] 406. The server determines an image vector corresponding to the UI design image; and determines a snapshot vector set corresponding to the snapshot time series data; wherein the snapshot vector set includes a snapshot vector corresponding to each UI snapshot.

[0136] Exemplarily, the server can encode the UI design image through a preset vector encoder to obtain an image vector corresponding to the UI design image; and based on the preset vector encoder, encode each UI snapshot in the snapshot time series data to obtain a snapshot vector corresponding to each UI snapshot in the vector set.

[0137] In one example, step 406 includes:

[0138] The first step is to determine a first three-channel histogram corresponding to the UI design image, and to determine a second three-channel histogram corresponding to the UI snapshot; wherein the first three-channel histogram includes first color information of each first pixel in the UI design image, and the first color information includes the color value of the first pixel under each preset color channel; the second three-channel histogram includes second color information of each second pixel in the UI snapshot, and the second color information includes the color value of the second pixel under each preset color channel.

[0139] The second step is to determine the image vector corresponding to the UI design image according to the first three-channel histogram; and to determine the snapshot vector corresponding to the UI snapshot according to the second three-channel histogram.

[0140] Exemplarily, in the vector generation process, the server performs color recognition processing on the UI design image based on the image color processing module to obtain a first three-channel histogram corresponding to the UI design image, wherein the first three-channel histogram includes the color value of each first pixel in the UI design image under each preset color channel, i.e., the first color information; at the same time, based on the image color processing module, color recognition processing is performed on each UI snapshot to obtain a second three-channel histogram corresponding to each UI snapshot, wherein each second three-channel histogram includes the color value of each second pixel in each UI snapshot under each preset color channel, i.e., the second color information. Among them, the preset color channel is a red channel, a green channel, or a blue channel. Based on the image vector encoder of the server, vector encoding processing is performed on the first color information of each first pixel in the first three-channel histogram corresponding to the UI design image to obtain the image vector corresponding to the UI design image; at the same time, based on the image vector encoder, vector encoding processing is performed on the second color information of each second pixel in the second three-channel histogram corresponding to each UI snapshot to obtain the snapshot vector corresponding to each UI snapshot.

[0141] In one example, the first step includes:

[0142] Step 1: perform partition combination processing on at least one preset color value corresponding to each preset color channel to obtain at least one combined partition; wherein the combined partition includes a color partition corresponding to each preset color channel; the color partition includes a preset number of continuous preset color values ​​under each preset color channel; and the preset color values ​​in each color partition corresponding to the preset color channel do not overlap with each other.

[0143] Step 2: According to the first color information, determine the first combined partition corresponding to the first pixel from each combined partition; wherein the first combined partition is a combined partition in which the color partition under each preset color channel is consistent with the first color partition corresponding to the first pixel under the preset color channel; the first color partition is a color partition in which a preset color value that is consistent with the color value under the preset color channel in the first color information is located.

[0144] Step three, according to the second color information, determine the second combined partition corresponding to the second pixel point from each combined partition; wherein the second combined partition is a combined partition in which the color partition under each preset color channel is consistent with the second color partition corresponding to the second pixel point under the preset color channel; the second color partition is a color partition where the preset color value is consistent with the color value under the preset color channel in the second color information.

[0145] Step 4: Concatenate the first numbers corresponding to each combined partition to obtain an image vector corresponding to the UI design image; and for each UI snapshot, concatenate the second numbers corresponding to each combined partition to obtain a snapshot vector corresponding to the UI snapshot; wherein the first number is the total number of all first pixel points corresponding to the combined partition; and the second number is the total number of all second pixel points corresponding to each combined partition in each UI snapshot.

[0146] Specifically, based on the server, for each preset color channel, such as each preset color channel of red R, green G, and blue B, all preset color values ​​under each preset color channel are first partitioned to obtain multiple color partitions corresponding to each preset color channel, each color partition includes a preset number of continuous preset color values ​​under each preset color channel, and the preset color values ​​in each color partition corresponding to each preset color channel do not repeat each other. The color partitions corresponding to each preset color channel are combined to obtain multiple combined partitions, each combined partition includes a color partition corresponding to each preset color channel. According to the color information of each pixel in the first three-channel histogram corresponding to the UI design image, that is, the first color information, first for each preset color channel, determine the color partition where the preset color value is consistent with the color value under each preset color channel in the first color information, which is the color partition corresponding to each pixel in the first three-channel histogram under each preset color channel, that is, the first color partition; the combined partition where the color partition under each preset color channel is consistent with the first color partition corresponding to each preset color channel is determined as the combined partition corresponding to each pixel in the first three-channel histogram, that is, the first combined partition. At the same time, for each pixel in the second three-channel histogram corresponding to each UI snapshot, that is, the second color information, first determine the color partition where the preset color value consistent with the color value in each preset color channel in the second color information is located for each preset color channel, which is the color partition corresponding to each pixel in the second three-channel histogram under each preset color channel, that is, the second color partition; determine the combined partition where the color partition under each preset color channel is consistent with the second color partition corresponding to each preset color channel as the combined partition corresponding to each pixel in the second three-channel histogram, that is, the second combined partition. For all pixels in the UI design image, determine the total number of pixels in the UI design image corresponding to each combined partition, that is, the first number of each combined partition, and splice the first numbers corresponding to each combined partition to obtain the image vector corresponding to the UI design image; at the same time, for all pixels in each UI snapshot, determine the total number of pixels in the UI snapshot corresponding to each combined partition, that is, the second number of each combined partition corresponding to each UI snapshot, and splice the second number of each combined partition corresponding to each UI snapshot to obtain the snapshot vector corresponding to each UI snapshot.

[0147] For example, Figure 6 A flowchart of a user interface testing solution provided in an embodiment of the present application is shown in FIG. Figure 6As shown, for each channel of red R, green G, and blue B, each channel has 256 values, and each RGB channel can be divided into 4 color partitions: 0-63 (zone 0), 64-127 (zone 1), 128-191 (zone 2), and 192-255 (zone 3). Each zone of each RGB channel is combined to obtain 64 combination partitions, and each color belongs to the 64 combination partitions. For the color information of pixel 1 in the UI design image is (62, 254, 172), and the color information of pixel 2 is (30, 30, 30), then the combination partition where the pixel 1 is located is (0, 3, 2), and the combination partition where the pixel 2 is located is (0, 3, 2). The partition is (0, 0, 0), until the last pixel n (255, 255, 255) is determined to be located in the combined partition (3, 3, 3). For the UI design image, the number of pixels in each combined partition in all combined partitions, for example, the number of pixels in the combined partition (0, 0, 0), the combined partition (0, 0, 1), the combined partition (0, 0, 2), ..., the combined partition (3, 3, 3) are 3, 7, 4, ..., 5 respectively. Then, the number of pixels in each combined partition is spliced ​​to obtain the image vector of the UI design image as a 64-dimensional vector (3, 7, 4, ..., 5); then, the snapshot vector of each UI snapshot is calculated by this method. Among them, the number of color partitions can be adjusted, but the more the number of color partitions, the more complicated the calculation.

[0148] 407. The server determines the cosine similarity between the image vector and the snapshot vector.

[0149] Exemplarily, the server performs calculation processing on the image vector and each snapshot vector based on the cosine similarity calculation algorithm to obtain the cosine similarity between the image vector and each snapshot vector; for example, the formula Calculate and obtain the cosine similarity cosθ between the image vector and each snapshot vector; where A i is the vector value of the i-th dimension in the image vector, B i is the vector value of the i-th dimension in the snapshot vector.

[0150] 408. The server determines that the UI snapshot corresponding to the snapshot vector whose cosine similarity value is closest to 1 is a similar UI snapshot.

[0151] Exemplarily, the server determines the cosine similarity with the value closest to 1 from among the cosine similarities based on the calculated cosine similarity between the image vector and each snapshot vector, and uses the UI snapshot corresponding to the snapshot vector corresponding to the cosine similarity as the similar UI snapshot corresponding to the image vector.

[0152] 409. The server determines a first element feature in the image features of the UI design image; and determines a second element feature in the image features of the similar UI snapshot; wherein the first element feature is an element feature of each page element in the UI design image; and the second element feature is an element feature of each page element in the similar UI snapshot.

[0153] Exemplarily, the server extracts the image features of the UI design image based on the image feature extraction algorithm, and extracts the element features of each page element in the UI design image, i.e., the first element features; at the same time, based on the image feature extraction algorithm, the server extracts the image features of the UI snapshot-like image, and extracts the element features of each page element in the UI snapshot-like image, i.e., the second element features. The element features may include, but are not limited to, feature description data such as the position, size, and color of the relevant elements.

[0154] In one example, step 409 includes:

[0155] The first step is to perform image segmentation processing on the UI design image and the similar UI snapshot respectively to obtain at least one first graphical user interface GUI element picture corresponding to the UI design image and at least one second GUI element picture corresponding to the similar UI snapshot.

[0156] The second step is to perform feature processing on each first GUI element image to obtain first element features; and perform feature processing on each second GUI element image to obtain second element features.

[0157] Exemplarily, based on the target detection algorithm, the server identifies at least one first graphical user interface (GUI) element corresponding to the UI design image, and performs image segmentation processing to obtain at least one first GUI element image corresponding to the UI design image; at the same time, based on the target detection algorithm, it identifies at least one second GUI element corresponding to a similar UI snapshot, and performs image segmentation processing to obtain at least one second GUI element image corresponding to a similar UI snapshot. Based on the image processing technology, feature processing is performed on all first GUI element images to obtain first element features corresponding to the UI design image; at the same time, feature processing is performed on all second GUI element images to obtain second element features corresponding to similar UI snapshots.

[0158] In one example, the first step of step 409 includes:

[0159] Step 1: Perform edge detection on the UI design image and the similar UI snapshot respectively to obtain original contour information of the UI design image and snapshot contour information of the similar UI snapshot; wherein the original contour information includes at least one design contour image; the design contour image represents the background contour of the UI element in the UI design image; the snapshot contour information includes at least one snapshot contour image; the snapshot contour image represents the background contour of the UI element in the similar UI snapshot.

[0160] Step 2: Cut the design outline image to obtain a first GUI element image; and cut the snapshot outline image to obtain a second GUI element image.

[0161] Specifically, based on the edge detection algorithm, the server identifies each UI element in the UI design image and performs edge detection processing to obtain the background contour of each UI element in the UI design image, that is, the design contour image of each UI element. The design contour images of all UI elements in the UI design image constitute the original contour information; at the same time, based on the edge detection algorithm, each UI element in the similar UI snapshot is identified and edge detection processing is performed to obtain the background contour of each UI element in the similar UI snapshot, that is, the snapshot contour image of each UI element. The snapshot contour images of all UI elements in the similar UI snapshot constitute the snapshot contour information. Based on the image cutting technology, each design contour image of the UI design image is cross-cut and cross-cut to obtain each first GUI element image corresponding to the UI design image; at the same time, each snapshot contour image of the similar UI snapshot is cross-cut and cross-cut to obtain each second GUI element image corresponding to the similar UI snapshot.

[0162] For example, in a web page, the display of content has a hierarchical relationship, which is divided into background and foreground. First, it is necessary to find the connected area of ​​the background to detect the edge of the element. In order to prevent the display of content in the page, the background is mostly solid color or gradient color, which is relatively simple and has obvious edges. Therefore, the Laplacian operator is first used to perform edge detection on the design draft and UI snapshot to roughly extract the background contours of each element. The UI page is basically a vertical streaming layout, so it can be cut horizontally and vertically according to the results obtained in the previous step to obtain each GUI element image of the design draft and each GUI element image of the UI snapshot, such as input box images, drop-down box images, etc., and the coordinates of the cutting point of each design contour image of the UI design image during the cutting process in the UI design image, and the coordinates of the cutting point of each snapshot contour image of the similar UI snapshot during the cutting process in the similar UI snapshot, are recorded respectively for processing.

[0163] In one example, the first step of step 409 further includes: performing a flood filling process on the design contour image and the snapshot contour image to obtain a processed design contour image and a processed snapshot contour image.

[0164] For example, the result of edge detection processing has more noise and needs further processing of details. Use the flood filling algorithm, set appropriate seed nodes and threshold ranges, and perform flood filling processing on each design contour image and each snapshot contour image respectively, to obtain each processed design contour image and each processed snapshot contour image respectively, to achieve further connection of the background area and reduce noise.

[0165] In one example, the second step of step 409 includes:

[0166] Step 1: Perform target detection processing on the first GUI element image and the second GUI element image respectively to obtain first component data and second component data; wherein the first component data represents the foreground component in the first GUI element image; and the second component data represents the foreground component in the second GUI element image.

[0167] Step 2: Perform attribute extraction processing on the foreground component in each first GUI element image to obtain component attribute features in the first element features; and perform attribute extraction processing on the foreground component in each second GUI element image to obtain component attribute features in the second element features.

[0168] Step three: determine the element position feature in the first element feature according to the cutting point coordinates of each first GUI element image; and determine the element position feature in the second element feature according to the cutting point coordinates of each second GUI element image.

[0169] Specifically, based on the target detection processing algorithm, the server performs target detection processing on each first GUI element image in the UI design image, and detects and obtains the foreground component in each first GUI element image in the UI design image, that is, the first component data; at the same time, the server performs target detection processing on each second GUI element image in the similar UI snapshot, and detects and obtains the foreground component in each second GUI element image in the similar UI snapshot. Through the feature extraction algorithm, the server performs attribute feature extraction processing on the foreground component in each first GUI element image in the UI design image, and obtains the component attribute feature in the first element feature corresponding to the UI design image; at the same time, the server performs attribute feature extraction processing on the foreground component in each second GUI element image in the similar UI snapshot, and obtains the component attribute feature in the second element feature corresponding to the similar UI snapshot. After performing cross-cutting and longitudinal cutting processing on each design outline image of the UI design image and each snapshot outline image of the similar UI snapshot based on the image cutting technology, the cutting point of each design outline image of the UI design image during the cutting processing has the cutting point coordinates of the cutting point in the UI design image, and the cutting point of each snapshot outline image of the similar UI snapshot during the cutting processing has the cutting point coordinates of the cutting point in the similar UI snapshot image. Then, according to the cutting point coordinates of each first GUI element image in the UI design image, the element position feature in the first element feature corresponding to the UI design image can be obtained; at the same time, according to the cutting point coordinates of each second GUI element image in the similar UI snapshot, the element position feature in the second element feature corresponding to the similar UI snapshot can be obtained.

[0170] For example, the target detection algorithm is used to identify components in the foreground. Different companies have slight differences in page and component design styles. To enhance the accuracy of target detection, the pages and components of the internal system are selected as sample data, and the Faster R-CNN algorithm framework is trained to identify foreground components in GUI element images. The attributes of the identified components are extracted to generate domain-specific language (DSL) JSON data. The extracted content is shown in Table 1 below.

[0171] Table 1

[0172] property key property key width width Font size fontSize high height Font fontFamily Round Corners radius Font Color fontColor Border Width borderWidth Font alignment textAlign Border Type borderType Font Spacing letterSpace Border Color borderColor Line Height lineHeight shadow boxShadow Internal spacing padding Background color background

[0173] 410. The server generates a UI test result of the user equipment according to the first element feature and the second element feature.

[0174] Exemplarily, the server performs feature comparison processing on the first element features of the UI design image and the second element features of the similar UI snapshot based on a feature comparison algorithm, compares the differences in the style, attributes and other contents of each element, and finally obtains the result of the UI restoration test.

[0175] In one example, step 410 includes:

[0176] The first step is to generate a first feature tree corresponding to the UI design image according to the first element feature; and to generate a second feature tree corresponding to the similar UI snapshot according to the second element feature; wherein the first feature tree represents the element hierarchy of each UI element in the UI design image and the inclusion relationship between each UI element; and the second feature tree represents the hierarchy, attributes and inclusion relationship between each UI element in the similar UI snapshot.

[0177] The second step is to traverse and compare the first feature tree and the second feature tree to obtain a UI test result of the user device.

[0178] Specifically, based on the DSL TREE generation algorithm, the server parses the first element feature of the UI design image to generate a DSL TREE corresponding to the UI design image, i.e., a first feature tree, to characterize the element hierarchy of each UI element in the UI design image and the inclusion relationship between each UI element; at the same time, based on the DSL TREE generation algorithm, the server parses the second element feature of the similar UI snapshot to generate a DSL TREE corresponding to the similar UI snapshot, i.e., a second feature tree, to characterize the element hierarchy of each UI element in the similar UI snapshot and the inclusion relationship between each UI element. The server traverses the obtained first feature tree and second feature tree, and compares the differences in the element features of each element, thereby obtaining the UI restoration test result of the user device.

[0179] For example, Figure 7 A schematic diagram of a DSL JSON example provided in an embodiment of the present application, such as Figure 7 As shown, based on the previous analysis steps, the DSL JSON layout description of the UI design image and the similar UI snapshot, that is, the element features of each element, is obtained. According to the element hierarchy and inclusion relationship described in the DSL JSON, the corresponding DSL TREE is generated. Figure 8 A schematic diagram of a UI restoration test situation provided in an embodiment of the present application, such as Figure 8 As shown, the two DSL TREEs corresponding to the UI design image and the similar UI snapshot are traversed, and the differences in the style, attributes and other contents of each element are compared to finally obtain the result of the UI restoration test.

[0180] In this embodiment, based on the above embodiments, on the one hand, in the real running environment of the WEB application, the snapshot collection of the page is completed with the help of canvas technology, and the obtained restoration test results can include the impact of the model; on the other hand, image recognition technology is used to segment page elements, extract element features to generate a feature tree, and automatically complete the restoration test results by traversing the feature tree. As a result, the UI restoration test can be automatically tested without writing a test script, thereby lowering the threshold for users.

[0181] Fig. 9 A schematic diagram of the structure of an image-based user interface testing device provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, the device is applied to a server; the device includes:

[0182] The acquisition module 501 is used to receive the snapshot time series data sent by the user equipment and acquire the designed user interface UI image of the user equipment; wherein the UI design image is the UI image of the user equipment with the original design; the snapshot time series data includes at least one UI snapshot of the user equipment at a moment;

[0183] A determination module 502 is used to determine the UI snapshot with the highest similarity to the UI design image from the snapshot time series data, as a similar UI snapshot of the UI design image;

[0184] The generating module 503 is used to generate a UI test result of the user device according to the image features of the UI design image and the image features of the similar UI snapshot; wherein the UI test result is a UI restoration degree test result of the user device.

[0185] In a possible implementation, generation module 503 is specifically used to: determine a first element feature in the image features of a UI design image; and determine a second element feature in the image features of a similar UI snapshot; wherein the first element feature is an element feature of each page element in the UI design image; and the second element feature is an element feature of each page element in the similar UI snapshot; and generate a UI test result for a user device based on the first element feature and the second element feature.

[0186] In a possible implementation, the generation module 503 is specifically used to: perform image segmentation processing on the UI design image and the similar UI snapshot respectively to obtain at least one first graphical user interface GUI element image corresponding to the UI design image and at least one second GUI element image corresponding to the similar UI snapshot; perform feature processing on each first GUI element image to obtain a first element feature; and perform feature processing on each second GUI element image to obtain a second element feature.

[0187] In a possible implementation, the generation module 503 is specifically used to: perform edge detection on the UI design image and the similar UI snapshot respectively to obtain original contour information of the UI design image and snapshot contour information of the similar UI snapshot; wherein the original contour information includes at least one design contour image; the design contour image represents the background contour of the UI element in the UI design image; the snapshot contour information includes at least one snapshot contour image; the snapshot contour image represents the background contour of the UI element in the similar UI snapshot; perform cutting processing on the design contour image to obtain a first GUI element image; and perform cutting processing on the snapshot contour image to obtain a second GUI element image.

[0188] In a possible implementation manner, the generating module 503 is further specifically configured to: perform a flood filling process on the design contour image and the snapshot contour image to obtain a processed design contour image and a processed snapshot contour image.

[0189] In a possible implementation, the generation module 503 is further specifically used to: perform target detection processing on the first GUI element image and the second GUI element image respectively to obtain first component data and second component data; wherein the first component data represents the foreground component in the first GUI element image; the second component data represents the foreground component in the second GUI element image; perform attribute extraction processing on the foreground component data in each first GUI element image to obtain component attribute features in the first element feature; and perform attribute extraction processing on the foreground component data in each second GUI element image to obtain component attribute features in the second element feature; determine the element position features in the first element feature based on the cutting point coordinates of each first GUI element image; and determine the element position features in the second element feature based on the cutting point coordinates of each second GUI element image.

[0190] In a possible implementation, the generation module 503 is further specifically used to: generate a first feature tree corresponding to the UI design image based on the first element feature; and generate a second feature tree corresponding to the similar UI snapshot based on the second element feature; wherein the first feature tree represents the element hierarchy of each UI element in the UI design image and the inclusion relationship between each UI element; the second feature tree represents the hierarchy, attributes of each UI element and the inclusion relationship between each UI element in the similar UI snapshot; and the first feature tree and the second feature tree are traversed and compared to obtain the UI test result of the user device.

[0191] In a possible implementation, the determination module 502 is specifically used to: determine the image vector corresponding to the UI design image; and determine the snapshot vector set corresponding to the snapshot time series data; wherein the snapshot vector set includes the snapshot vector corresponding to each UI snapshot; determine the cosine similarity between the image vector and the snapshot vector; and determine that the UI snapshot corresponding to the snapshot vector whose cosine similarity value is closest to 1 is a similar UI snapshot.

[0192] In a possible implementation, the determination module 502 is specifically used to: determine a first three-channel histogram corresponding to the UI design image, and determine a second three-channel histogram corresponding to the UI snapshot; wherein the first three-channel histogram includes first color information of each first pixel in the UI design image, and the first color information includes the color value of the first pixel under each preset color channel; the second three-channel histogram includes second color information of each second pixel in the UI snapshot, and the second color information includes the color value of the second pixel under each preset color channel; determine the image vector corresponding to the UI design image according to the first three-channel histogram; and determine the snapshot vector corresponding to the UI snapshot according to the second three-channel histogram.

[0193] In a possible implementation, the determination module 502 is specifically used to: perform partition combination processing on at least one preset color value corresponding to each preset color channel to obtain at least one combined partition; wherein the combined partition includes a color partition corresponding to each preset color channel; the color partition includes a preset number of continuous preset color values ​​under each preset color channel; the preset color values ​​in each color partition corresponding to the preset color channel are not repeated; according to the first color information, determine the first combined partition corresponding to the first pixel point from each combined partition; wherein the first combined partition is a combined partition in which the color partition under each preset color channel is consistent with the first color partition corresponding to the first pixel point under the preset color channel; the first color partition is a combined partition in which the color partition under each preset color channel is consistent with the first color partition corresponding to the first pixel point under the preset color channel; the first color partition is a combined partition in which the preset color value is consistent with the color value under the preset color channel in the first color information. in the color partition; according to the second color information, determining the second combined partition corresponding to the second pixel point from each combined partition; wherein the second combined partition is a combined partition in which the color partition under each preset color channel is consistent with the second color partition corresponding to the second pixel point under the preset color channel; the second color partition is a color partition where a preset color value consistent with the color value under the preset color channel in the second color information is located; the first number corresponding to each combined partition is spliced ​​to obtain an image vector corresponding to the UI design image; and for each UI snapshot, the second number corresponding to each combined partition is spliced ​​to obtain a snapshot vector corresponding to the UI snapshot; wherein the first number is the total number of all first pixel points corresponding to the combined partition; the second number is the total number of all second pixel points corresponding to each combined partition in each UI snapshot.

[0194] The device of this embodiment can execute the technical solution in the above method. Its specific implementation process and technical principles are the same and will not be repeated here.

[0195] Fig.10 A structural diagram of another image-based user interface testing device provided in an embodiment of the present application, such as Fig.10 As shown, the device is applied to user equipment; the device includes:

[0196] The drawing module 601 is used to create an initial drawing element, and draw each page element in the user interface UI of the user device at the current moment into the initial drawing element to obtain a drawing element;

[0197] The generating module 602 is used to perform data export processing on the drawing elements, generate a UI snapshot of the user device at the current moment, and add the UI snapshot to the snapshot time series data;

[0198] The sending module 603 is used to send the snapshot time series data to the server; wherein the server includes the device as described in the third aspect.

[0199] In a possible implementation, the drawing module 601 is specifically used to: parse the page currently displayed by the user device through a preset document object model DOM tree to obtain an element set; wherein the element set includes the style and layout information of each page element in the user interface UI of the user device; according to the DOM tree and the element set, draw each page element in the element set into the initial drawing element to obtain the drawing element.

[0200] In a possible implementation, the device is also used to: if it is determined that the number of snapshots in the snapshot time series data is greater than or equal to a preset number, stop document object parsing processing on the user interface UI of the user device; wherein the number of snapshots is the total number of UI snapshots in the snapshot time series data.

[0201] The device of this embodiment can execute the technical solution in the above method. Its specific implementation process and technical principles are the same and will not be repeated here.

[0202] Fig.11 A schematic diagram of the structure of a server provided in an embodiment of the present application, such as Fig.11 As shown, the server includes: a memory 701 and a processor 702; the memory 701 is a memory for storing instructions executable by the processor 702.

[0203] The processor 702 is configured to execute the method provided in the above embodiment.

[0204] The server further includes a receiver 703 and a transmitter 704. The receiver 703 is used to receive instructions and data sent by other devices, and the transmitter 704 is used to send instructions and data to external devices.

[0205] Fig.12 A schematic diagram of the structure of a user equipment provided in an embodiment of the present application, such as Fig.12 As shown, the user equipment includes: a memory 801 and a processor 802; the memory 801 is a memory for storing instructions executable by the processor 802.

[0206] The processor 802 is configured to execute the method provided in the above embodiment.

[0207] The user equipment further includes a receiver 803 and a transmitter 804. The receiver 803 is used to receive instructions and data sent by other devices, and the transmitter 504 is used to send instructions and data to external devices.

[0208] The specific implementation process of the processor can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0209] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0210] An embodiment of the present application also provides a chip for executing instructions, which is used to execute the technical solution of the processing method in the above embodiment.

[0211] The embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed on a computer, the computer executes the technical solution of the processing method of the above embodiment.

[0212] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0213] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in a dedicated integrated circuit. Of course, the processor and the readable storage medium can also exist in a device as discrete components.

[0214] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, the technical solution of the processing method in the above embodiment can be implemented.

[0215] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0216] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0217] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0218] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0219] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. An image-based user interface testing method, characterized in that: The method is applied to a server; the method comprises: Receive snapshot time series data sent by a user device, and obtain a user interface UI design image of the user device; wherein the UI design image is an originally designed UI image of the user device; the snapshot time series data includes a UI snapshot of the user device at at least one moment; Determine, from the snapshot time series data, a UI snapshot having the highest similarity to the UI design image as a similar UI snapshot of the UI design image; A UI test result of the user device is generated according to the image features of the UI design image and the image features of the similar UI snapshot; wherein the UI test result is a UI restoration degree test result of the user device.

2. The method according to claim 1, characterized in that Generating a UI test result of the user device according to the image feature of the UI design image and the image feature of the similar UI snapshot, including: Determine a first element feature in the image features of the UI design image; and determine a second element feature in the image features of the similar UI snapshot; wherein the first element feature is an element feature of each page element in the UI design image; and the second element feature is an element feature of each page element in the similar UI snapshot; Generate a UI test result of the user equipment according to the first element feature and the second element feature.

3. The method according to claim 2, characterized in that Determine a first element feature among the image features of the UI design image; And determining a second element feature in the image feature of the similar UI snapshot, including: Performing image segmentation processing on the UI design image and the similar UI snapshot respectively to obtain at least one first graphical user interface GUI element picture corresponding to the UI design image and at least one second GUI element picture corresponding to the similar UI snapshot; Perform feature processing on each of the first GUI element images to obtain the first element features; and perform feature processing on each of the second GUI element images to obtain the second element features.

4. The method according to claim 3, characterized in that Performing image segmentation processing on the UI design image and the similar UI snapshot respectively to obtain at least one first graphical user interface GUI element picture corresponding to the UI design image and at least one second GUI element picture corresponding to the similar UI snapshot, including: Performing edge detection on the UI design image and the similar UI snapshot respectively, to obtain original contour information of the UI design image and snapshot contour information of the similar UI snapshot; wherein the original contour information includes at least one design contour image; the design contour image represents the background contour of the UI element in the UI design image; the snapshot contour information includes at least one snapshot contour image; the snapshot contour image represents the background contour of the UI element in the similar UI snapshot; The design outline image is cut to obtain the first GUI element image; and the snapshot outline image is cut to obtain the second GUI element image.

5. The method according to claim 4, characterized in that The method further comprises: The design contour image and the snapshot contour image are subjected to a flood filling process to obtain a processed design contour image and a processed snapshot contour image.

6. The method according to claim 3, characterized in that Performing feature processing on each of the first GUI element images to obtain the first element features; And performing feature processing on each of the second GUI element images to obtain the second element features, including: Performing target detection processing on the first GUI element image and the second GUI element image respectively to obtain first component data and second component data; wherein the first component data represents a foreground component in the first GUI element image; and the second component data represents a foreground component in the second GUI element image; Performing attribute extraction processing on the foreground component data in each of the first GUI element images to obtain component attribute features in the first element features; and performing attribute extraction processing on the foreground component data in each of the second GUI element images to obtain component attribute features in the second element features; According to the cutting point coordinates of each of the first GUI element images, the element position features in the first element features are determined; and according to the cutting point coordinates of each of the second GUI element images, the element position features in the second element features are determined.

7. The method according to claim 2, characterized in that Generating a UI test result of the user equipment according to the first element feature and the second element feature includes: Generate a first feature tree corresponding to the UI design image according to the first element feature; and generate a second feature tree corresponding to the similar UI snapshot according to the second element feature; wherein the first feature tree represents the element hierarchy of each UI element in the UI design image and the inclusion relationship between each UI element; and the second feature tree represents the hierarchy, attributes and inclusion relationship between each UI element in the similar UI snapshot; The first feature tree and the second feature tree are traversed and compared to obtain a UI test result of the user equipment.

8. The method according to any one of claims 1 to 7, characterized in that Determining a UI snapshot having the highest similarity to the UI design image from the snapshot time series data as a similar UI snapshot of the UI design image includes: Determine an image vector corresponding to the UI design image; and determine a snapshot vector set corresponding to the snapshot time series data; wherein the snapshot vector set includes a snapshot vector corresponding to each of the UI snapshots; determining a cosine similarity between the image vector and the snapshot vector; Determine the UI snapshot corresponding to the snapshot vector whose cosine similarity value is closest to 1 as the similar UI snapshot.

9. The method according to claim 8, characterized in that Determining an image vector corresponding to the UI design image; and determining a set of snapshot vectors corresponding to the snapshot time series data, including: Determine a first three-channel histogram corresponding to the UI design image, and determine a second three-channel histogram corresponding to the UI snapshot; wherein the first three-channel histogram includes first color information of each first pixel in the UI design image, and the first color information includes a color value of the first pixel in each preset color channel; the second three-channel histogram includes second color information of each second pixel in the UI snapshot, and the second color information includes a color value of the second pixel in each preset color channel; An image vector corresponding to the UI design image is determined according to the first three-channel histogram; and a snapshot vector corresponding to the UI snapshot is determined according to the second three-channel histogram.

10. The method according to claim 9, characterized in that Determining an image vector corresponding to the UI design image according to the first three-channel histogram; and determining a snapshot vector corresponding to the UI snapshot according to the second three-channel histogram, including: Performing a partition combination process on at least one preset color value corresponding to each of the preset color channels to obtain at least one combined partition; wherein the combined partition includes a color partition corresponding to each of the preset color channels; the color partition includes a preset number of continuous preset color values ​​under each of the preset color channels; the preset color values ​​in each color partition corresponding to the preset color channel do not overlap with each other; According to the first color information, a first combined partition corresponding to the first pixel is determined from each of the combined partitions; wherein the first combined partition is a combined partition in which the color partition under each of the preset color channels is consistent with the first color partition corresponding to the first pixel under the preset color channel; the first color partition is a color partition in which a preset color value consistent with the color value under the preset color channel in the first color information is located; According to the second color information, a second combined partition corresponding to the second pixel is determined from each of the combined partitions; wherein the second combined partition is a combined partition in which the color partition under each of the preset color channels is consistent with the second color partition corresponding to the second pixel under the preset color channel; and the second color partition is a color partition in which a preset color value consistent with the color value under the preset color channel in the second color information is located; The first numbers corresponding to each of the combined partitions are spliced ​​together to obtain an image vector corresponding to the UI design image; and for each of the UI snapshots, the second numbers corresponding to each of the combined partitions are spliced ​​together to obtain a snapshot vector corresponding to the UI snapshot; wherein the first number is the total number of all first pixel points corresponding to the combined partition; and the second number is the total number of all second pixel points corresponding to each combined partition in each UI snapshot.

11. An image-based user interface testing method, characterized in that: The method is applied to a user equipment; the method comprises: Creating an initial drawing element, and drawing each page element in the user interface UI of the user device at the current moment into the initial drawing element to obtain a drawing element; Performing data export processing on the drawing element to generate a UI snapshot of the user device at the current moment; and adding the UI snapshot to the snapshot time series data; The snapshot time series data is sent to a server; wherein the server is used to execute the method as described in any one of claims 1-10.

12. The method according to claim 11, characterized in that Drawing each page element in the user interface UI of the user device at the current moment into the initial drawing element to obtain a drawing element, including: The page currently displayed by the user device is parsed through a preset document object model DOM tree to obtain an element set; wherein the element set includes the style and layout information of each page element in the user interface UI of the user device; According to the DOM tree and the element set, each page element in the element set is drawn into the initial drawing element to obtain the drawing element.

13. The method according to claim 11 or 12, characterized in that: The method further comprises: If it is determined that the number of snapshots in the snapshot time series data is greater than or equal to a preset number, the document object parsing process for the user interface UI of the user device is stopped; wherein the number of snapshots is the total number of UI snapshots in the snapshot time series data.

14. An image-based user interface testing device, characterized in that: The device is applied to a server; the device comprises: An acquisition module, configured to receive snapshot time series data sent by a user device, and acquire a designed user interface UI image of the user device; wherein the UI design image is an originally designed UI image of the user device; and the snapshot time series data includes a UI snapshot of the user device at at least one moment; a determination module, configured to determine, from the snapshot time series data, a UI snapshot having the highest similarity to the UI design image, as a similar UI snapshot of the UI design image; A generating module is used to generate a UI test result of the user device according to the image features of the UI design image and the image features of the similar UI snapshot; wherein the UI test result is a UI restoration test result of the user device.

15. An image-based user interface testing device, characterized in that: The device is applied to user equipment; the device includes: A drawing module, used to create an initial drawing element, and draw each page element in the user interface UI of the user device at the current moment into the initial drawing element to obtain a drawing element; A generating module, configured to perform data export processing on the drawing element, generate a UI snapshot of the user device at the current moment; and add the UI snapshot to the snapshot time series data; A sending module, used for sending the snapshot time series data to a server; wherein the server includes the device as claimed in claim 14.

16. A server, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 10.

17. A user equipment, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 11 to 13.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 13 when executed by a processor.

19. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 13 when being executed by a processor.

Images