UI compatibility detection method and device

Abstract

The present application discloses a UI compatibility detection method and device, which comprises: obtaining UI hierarchical structure information when multiple devices play the same page image to be tested through a test tool interface; obtaining first node information based on the UI hierarchical structure information of a reference device; obtaining second node information based on the UI hierarchical structure information of the device to be tested; determining abnormal nodes based on the first node information and the second node information; determining the display position of abnormal nodes in the test tool interface based on the screen size information of the device to be tested, the image size information of the page image to be tested, and the interface size information of the test tool interface; generating abnormal annotations in the test tool interface according to the display position of abnormal nodes. The method utilizes the UI hierarchical structure information of multiple devices to identify abnormal nodes, and can eliminate detection errors caused by different types and resolutions between multiple devices in the prior art, thereby improving the accuracy of UI compatibility detection.

Description

Technical Field

[0001] The present application relates to the field of compatibility testing, and in particular to a UI compatibility testing method and device. Background Art

[0002] Before the video playback software is launched, UI compatibility testing needs to be performed for different types of devices (e.g., different manufacturers, different operating systems). Existing UI compatibility testing methods usually use image recognition, that is, after determining the screen of the reference device as the reference image, the screen of the device to be tested is compared with the reference image one by one. Due to the different resolutions of various devices and the dynamic screen of the video playback software, it is easy to cause errors in the image recognition results, and thus it is impossible to obtain accurate UI compatibility test results. Summary of the invention

[0003] The present application provides a UI compatibility detection method and device, the purpose of which is to improve the accuracy of UI compatibility detection.

[0004] In order to achieve the above objectives, this application provides the following technical solutions:

[0005] A UI compatibility detection method, comprising:

[0006] Obtaining UI hierarchical structure information when multiple devices play the same page image to be tested through a test tool interface; the multiple devices include a reference device and a device to be tested;

[0007] Based on the UI hierarchical structure information of the reference device, obtaining corresponding first node information;

[0008] Based on the UI hierarchical structure information of the device under test, obtaining corresponding second node information;

[0009] Based on the first node information and the second node information, an abnormal node is determined; the abnormal node is used to represent an abnormal UI element that is different between the device under test and the reference device when playing the same page image under test;

[0010] Determine the display position of the abnormal node in the test tool interface based on the screen size information of the device to be tested, the image size information of the page image to be tested, and the interface size information of the test tool interface;

[0011] According to the display position of the abnormal node, a corresponding abnormal annotation is generated in the test tool interface.

[0012] Optionally, determining an abnormal node based on the first node information and the second node information includes:

[0013] Based on the first node information, determining a first node set; the first node set includes a plurality of first nodes and corresponding attributes;

[0014] Based on the second node information, determining a second node set; the second node set includes a plurality of second nodes and corresponding attributes;

[0015] According to corresponding attributes, a plurality of the first nodes and a plurality of the second nodes are matched to obtain matched points and unmatched points; the matched points represent second nodes that are successfully matched, and the unmatched points represent second nodes that fail to match;

[0016] Determining whether the attribute of the matched point is abnormal according to the attribute of the first node matched with the matched point;

[0017] The matched points with abnormal attributes and the unmatched points are determined as abnormal nodes.

[0018] Optionally, if the attribute of the first node is the same as the attribute of the second node, it is determined that the first node matches the second node; if the attribute of the first node is different from the attribute of the second node, it is determined that the first node does not match the second node.

[0019] Optionally, determining whether the attribute of the matched point is abnormal according to the attribute of the first node that matches the matched point includes:

[0020] Obtaining first screen size information and second screen size information; the first screen size information is the screen size information of the reference device, and the second screen size information is the screen size information of the device under test;

[0021] Determine estimated node coordinates based on the first picture size information, the second picture size information, and an attribute of a first node that matches the matched point;

[0022] According to the estimated node coordinates, it is determined whether the attributes of the matched points are abnormal.

[0023] Optionally, determining the estimated node coordinates based on the first picture size information, the second picture size information, and an attribute of a first node that matches the matched point includes:

[0024] Based on the first picture size information, obtaining a first width value and a first height value;

[0025] Based on the second picture size information, obtaining a second width value and a second height value;

[0026] Based on the attribute of the first node that matches the matched point, obtain the node coordinates [x1, y1] [x2, y2] of the first node that matches the matched point;

[0027] Based on the first width value, the first height value, the second width value, the second height value, and the node coordinates [x1, y1][x2, y2], the estimated node coordinates [x3, y3][x4, y4] are calculated; wherein, x3=x1×(second width value / first width value), y3=y1×(second height value / first height value), x4=x2×(second width value / first width value), y4=y2×(second height value / first height value).

[0028] Optionally, determining whether the attribute of the matched point is abnormal according to the estimated node coordinates includes:

[0029] Based on the attributes of the matched points, obtaining the node coordinates of the matched points;

[0030] Determining the error between the estimated node coordinates and the node coordinates of the matched points;

[0031] If the error meets the preset requirement, determining that the attribute of the matched point is normal;

[0032] If the error does not meet the preset requirement, it is determined that the attribute of the matched point is abnormal.

[0033] Optionally, determining a display position of the abnormal node in the test tool interface based on the screen size information of the device to be tested, the image size information of the page image to be tested, and the interface size information of the test tool interface includes:

[0034] Based on the screen size information of the device under test, obtaining a second width value and a second height value;

[0035] Based on the image size information of the page image to be tested, obtaining a third width value and a third height value;

[0036] Based on the interface size information of the test tool interface, obtaining a fourth width value and a fourth height value;

[0037] Based on the attributes of the abnormal node, obtain the node coordinates [x5, y5] [x6, y6] of the abnormal node;

[0038] Based on the second width value, the second height value, the third width value, the third height value, the fourth width value, the fourth height value, and the node coordinates [x5, y5][x6, y6], the target node coordinates [x7, y7][x8, y8] are calculated; wherein, x7=x5×(third width value / second width value)+(fourth width value-third width value) / 2, y7=y5×(third height value / second height value)+(fourth height value-third height value) / 2, x8=x6×(third width value / second width value)+(fourth width value-third width value) / 2, y8=y6×(third height value / second height value)+(fourth height value-third height value) / 2;

[0039] Based on the target node coordinates [x7, y7] [x8, y8], the display position of the abnormal node in the test tool interface is determined.

[0040] A UI compatibility detection device, comprising:

[0041] A structure information collection unit, used to obtain UI hierarchical structure information when multiple devices play the same page image to be tested through the test tool interface; the multiple devices include a reference device and a device to be tested;

[0042] A first node analysis unit, configured to obtain corresponding first node information based on UI hierarchical structure information of the reference device;

[0043] A second node analysis unit, configured to obtain corresponding second node information based on the UI hierarchical structure information of the device under test;

[0044] an abnormal node identification unit, configured to determine an abnormal node based on the first node information and the second node information; the abnormal node is used to represent an abnormal UI element that is different between the device under test and the reference device when playing the same page image under test;

[0045] an abnormal position identification unit, configured to determine a display position of the abnormal node in the test tool interface based on the screen size information of the device to be tested, the image size information of the page image to be tested, and the interface size information of the test tool interface;

[0046] The abnormal annotation generating unit is used to generate a corresponding abnormal annotation in the test tool interface according to the display position of the abnormal node.

[0047] A storage medium includes a stored program, wherein the program executes the UI compatibility detection method when executed by a processor.

[0048] An electronic device, comprising: a processor, a memory and a bus; the processor and the memory are connected via the bus;

[0049] The memory is used to store programs, and the processor is used to run programs, wherein the program executes the UI compatibility detection method when the processor runs it.

[0050] The technical solution provided by the present application obtains the UI hierarchical structure information when multiple devices play the same page image to be tested through the test tool interface. Based on the UI hierarchical structure information of the reference device, obtain the first node information. Based on the UI hierarchical structure information of the device to be tested, obtain the second node information. Based on the first node information and the second node information, determine the abnormal node. Based on the screen size information of the device to be tested, the image size information of the page image to be tested, and the interface size information of the test tool interface, determine the display position of the abnormal node in the test tool interface. Generate an abnormal annotation in the test tool interface according to the display position of the abnormal node. The present application utilizes the UI hierarchical structure information of multiple devices to identify abnormal nodes, which can eliminate the detection errors caused by different types and resolutions between multiple devices in the prior art, thereby improving the accuracy of UI compatibility detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 A flowchart of a UI compatibility detection method provided in an embodiment of the present application;

[0053] Figure 2 A flowchart of another UI compatibility detection method provided in an embodiment of the present application;

[0054] Figure 3 A flowchart of another UI compatibility detection method provided in an embodiment of the present application;

[0055] Figure 4 A flowchart of another UI compatibility detection method provided in an embodiment of the present application;

[0056] Figure 5 A flowchart of another UI compatibility detection method provided in an embodiment of the present application;

[0057] Figure 6A schematic diagram of the architecture of a UI compatibility detection device provided in an embodiment of the present application;

[0058] Figure 7 A network topology diagram provided for an embodiment of the present application;

[0059] Figure 8 A schematic diagram of an interface provided in an embodiment of the present application;

[0060] Fig. 9 A schematic diagram of UI hierarchical structure information provided in an embodiment of the present application;

[0061] Fig.10 A schematic diagram of a UI compatibility detection effect provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0063] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0064] like Figure 1 As shown, it is a flow chart of a UI compatibility detection method provided in an embodiment of the present application, including the following steps.

[0065] S101: Obtain UI hierarchical structure information when multiple devices play the same page image to be tested through a test tool interface.

[0066] The multiple devices include a reference device and a device under test.

[0067] In some examples, the types of the reference device and the device under test may be Android devices (eg, set-top boxes). In possible implementations, the number of the devices under test may be one or more.

[0068] It should be noted that for test scenarios with multiple devices to be tested, the following steps are required: test device connection, create Android service, obtain and display device images, control devices in batches using keyboard, and UI compatibility detection.

[0069] In some examples, for the implementation process of the test device connection, multiple devices under test and reference devices can be connected to the test device (the test device includes a large screen, and the display interface of the large screen can be regarded as the test tool interface) through wired or wireless connection. Specifically, the type of the test device can be PC (Personal Computer).

[0070] In a possible implementation manner, the test device, the device under test, and the reference device may form a Figure 7 In the same LAN, Android devices (i.e. set-top boxes) can communicate with PCs via ADB (Android Debug Bridge) service in wired or wireless manner. After the PC and set-top boxes are connected normally, relevant commands can be sent to control each set-top box.

[0071] The so-called ADB is a command-line tool used by Android developers that allows them to communicate with a connected Android device or a running Android emulator. The ADB service is a daemon on an Android device that handles requests from ADB clients on a computer.

[0072] In some examples, the creation of Android services is mainly used to enable the device under test and the reference device to simultaneously respond to the test commands entered by the tester to perform corresponding operations. The process of creating an Android service can be summarized as follows: (1) Start the server: obtain all Android devices connected to the PC, push the mgtv-control-server.jar file to the directory of all connected Android devices through the adbpush command, and use the adbshell command combined with app_process to start mgtv-control-server.jar, for example: adbshellCLASSPATH= / data / local / tmp / mgtv-control-server.jarapp_process / com.mgtv.control.Server; (2) Establish a socket connection: After mgtv-control-server.jar is started, two LocalSockets are created, one for video streaming and the other for device control. The video stream socket is used to transmit the encoded screen video data to the PC, and the control socket is used to receive control instructions from the PC, such as keyboard. and mouse events; (3) Screen capture and encoding: mgtv-control-server.jar uses Android's MediaCodecAPI to encode the screen content. The encoding process is managed by the ScreenEncoder class, which creates an input Surface and associates it with the device's display Surface. It then starts the encoding process and sends the encoded data to the PC through the video stream socket; (4) Processing input events: When the PC client receives keyboard or mouse events, these events are sent to mgtv-control-server.jar through the control socket. mgtv-control-server.jar calls the injectInputEvent method of Android's InputManager through reflection to inject the input events into the Android system, thereby simulating user operations; (5) Data transmission: mgtv-control-server.jar continuously sends the captured data (such as the UI hierarchy structure information and screen size information of the device under test and the reference device) to the PC through LocalSocket, while listening to control commands from the PC to achieve two-way communication.

[0073] In a possible implementation, mgtv-control-server.jar is a server-side component of the compatibility test tool, which runs on the Android device and is responsible for capturing the device screen, processing input events, and transmitting the screen content to the PC in real time. app_process is a command used to run Java applications in the Android system, allowing mgtv-control-server.jar to run with shell privileges on the device.

[0074] In some examples, device image acquisition and display are mainly used to obtain page images played on the large screen of the device under test and the reference device. Specifically, the process of device image acquisition and display can be summarized as follows: (1) Create a client (i.e., a service program running UI compatibility detection on the PC) GUI (Graphical User Interface) canvas, obtain N devices (including the device under test and the reference device), and divide the canvas into N / 2 rows; (2) Establish a socket connection with mgtv-control-server.jar on all devices, and receive the video streams sent by all devices in real time; (3) Since each device may have different resolutions, it is necessary to unify the display screen size output by each device. At the same time, it is necessary to consider that the GUI window can be dynamically adjusted, that is, the screen needs to be scaled and stretched before rendering. The final style presentation effect can be seen in Figure 8 shown.

[0075] In a possible implementation, Figure 8 Taking the screen scaling process shown as an example, assuming that the number of devices obtained is 4, the parent page is divided equally according to the number of devices, and the parent page can be divided into 2 rows and 2 columns, a total of 4 sub-pages. For each device, the image size information (including image width value and image height value) of the page image output by the device is obtained, the ratio of the image width value and the image height value is calculated to obtain the first aspect ratio, the window information (including window width value and window height value) of the sub-page corresponding to the device is obtained, and the ratio of the window width value and the window height value is calculated to obtain the second aspect ratio. In order to ensure that the screen output by the device can be fully displayed on the large screen, the page image output by the device is dynamically adjusted according to the second aspect ratio, so that the second aspect ratio is greater than the first aspect ratio, the image height value = window height value, the image width value = image height value × first aspect ratio, or the image width value = window width value, and the image height value = image width value × first aspect ratio. Through continuous dynamic adjustment, the page images output by all devices are mapped and rendered on the parent page. For specific effects, see Figure 8In addition, since image rendering consumes a lot of CPU, the frequency of sending video frames on the server can be reduced, and the local client can increase the rendering interval time to reduce the CPU usage of the PC.

[0076] In a possible implementation, MediaCodec can provide the implementation function of device image acquisition. MediaCodec is an API for audio and video encoding and decoding provided by Android, allowing applications to encode and decode audio and video data. By using MediaCodec to encode the screen content, the encoder obtains input from the Surface associated with the display and outputs the encoded video stream.

[0077] Specifically, the execution process of MediaCodec can be summarized as follows: (1) Set the format of the video stream through the createFormat method, including parameters such as bit rate and frame rate; (2) Use the createCodec method to create an encoder instance and configure the encoder; (3) Use the createInputSurface method to create a Surface for encoder input; (4) Use the setDisplaySurface method to associate the encoder's input Surface with the device's display page; (5) After the encoder is started, the actual encoding process is performed through the encode method, and the encoded data is sent to the client through the socket; (6) The client creates a socket connection, receives the image sent by the server, and renders the received image to the local program.

[0078] In some examples, the process of keyboard batch control of devices can be summarized as follows: (1) The parent page binds the keyboard monitoring event. For example, taking the keyboard arrow key "up" as an example, when the tester clicks the keyboard arrow key "up", the keyboard event keysym='up' can be monitored; (2) Receive events sent from input devices (such as keyboards and mice), and convert the events into corresponding Android input events. For example, in the Android system, the up key code is 19, so 'up' needs to be converted into '19' that can be recognized by Android devices; (3) Receive keyboard events and perform corresponding conversions, and then send control events to all devices through the socket connections established on all devices; (4) All devices process according to the received control events to achieve the effect of controlling all devices.

[0079] In a possible implementation, the keyboard simulation operation can be implemented through the InputManagerService system service of Android. The InputManagerService is an Android system service responsible for processing various user input events, such as touch, key press, etc.

[0080] Specifically, the execution process of InputManagerService can be summarized as follows: (1) Listen to local keyboard events on the client. When the user presses or releases keys on the keyboard on the computer, these events are captured; (2) Encode keyboard events into a format that Android can understand, mainly converting keyboard key codes into corresponding key codes of Android; (3) Use Android's InputManager service to inject key events. Specifically, it injects key events into the Android system through the injectInputEvent method of InputManager; (4) The client creates a socket connection and sends keyboard events to the Android device to realize keyboard control of the device.

[0081] In some examples, based on the uiautomator function of AndroidSDKTools, the UI hierarchical structure information of the device under test and the reference device when they output page images can be captured. Specifically, after executing the adb command: adb shelluiautomator dump / data / local / tmp / dump.xml, the UI hierarchical structure information of the page image played on the test tool interface will be output to an XML file. Under normal circumstances, when the page image is scaled to the same size under different models, different systems, and different resolutions, the size and position of the same element are the same. Based on this principle, the UI hierarchical structure information of multiple devices playing the same page image under test on the test tool interface can be obtained through the adb command.

[0082] In some examples, the UI hierarchical structure information may represent the hierarchical relationship and organization between different elements and functions in a user interface (UI). Generally speaking, the UI hierarchical structure information includes multiple nodes of the device and corresponding attributes, as well as Fig. 9 As an example, the UI hierarchical structure information shown in the figure is used as an example. The so-called nodes can be found in Fig. 9 Node 1 and Node 2 are shown, and accordingly, the properties of the nodes include but are not limited to node coordinates (which can be based on Fig. 9 bounds as shown), and other parameters (such as index, which represents the order between sibling nodes).

[0083] In a possible implementation, the resolution of the device can be determined based on the root node of the UI hierarchical structure information (i.e., the first node in the XML file). Specifically, the attributes of the root node of the device include bounds (which can be regarded as node coordinates, expressed based on a combination of the upper left corner coordinates and the lower right corner coordinates), and bounds = "[0, 0][1920, 1080]", then the resolution of the device is 1920×1080.

[0084] S102: Based on the UI hierarchical structure information of the reference device, obtain corresponding first node information.

[0085] The first node information includes multiple first nodes of the reference device and corresponding attributes.

[0086] S103: Based on the UI hierarchical structure information of the device under test, obtain corresponding second node information.

[0087] The second node information includes multiple second nodes of the device under test and corresponding attributes.

[0088] S104: Determine an abnormal node based on the first node information and the second node information.

[0089] The abnormal node is used to represent abnormal UI elements that are different between the device under test and the reference device when playing the same page image under test.

[0090] It should be noted that the UI layout on the reference device is the correct template, and the abnormal UI element belongs to the device under test, that is, there is an abnormality in the UI layout of the device under test on the image of the page under test. The abnormality can be seen visually as deformation phenomena such as offset and stretching of UI elements (such as titles, introductions, feature film lists, etc.).

[0091] Optionally, the implementation process of determining the abnormal node based on the first node information and the second node information can be found in Figure 2 The steps are shown and the corresponding explanations.

[0092] S105: Determine a display position of an abnormal node in the test tool interface based on the screen size information of the device to be tested, the image size information of the page image to be tested, and the interface size information of the test tool interface.

[0093] Among them, the method for obtaining the screen size information of the device to be tested, the image size information of the page image to be tested, and the interface size information of the test tool interface can be referred to the explanation of step S101. Generally speaking, after determining the abnormal node, it is also necessary to mark the display position corresponding to the abnormal node on the test tool interface, and for this purpose, it is also necessary to determine the display position of the abnormal node in the test tool interface.

[0094] It should be noted that since the test device scales the images of the page to be tested output by multiple devices, the node coordinates of the abnormal node cannot be directly used as the display position (that is, the display position of the abnormal UI element corresponding to the abnormal node on the test tool interface is not the node coordinates of the abnormal node). It is necessary to use the screen size information of the device to be tested, the image size information of the page to be tested, and the interface size information of the test tool interface to convert the node coordinates of the abnormal node in order to obtain the display position of the abnormal node in the test tool interface.

[0095] Optionally, based on the screen size information of the device to be tested, the image size information of the page image to be tested, and the interface size information of the test tool interface, the implementation process of determining the display position of the abnormal node in the test tool interface can be seen in Figure 5 The steps are shown and the corresponding explanations.

[0096] S106: Generate a corresponding abnormal annotation in the test tool interface according to the display position of the abnormal node.

[0097] Among them, after determining the display position of the abnormal node, the corresponding abnormal annotation is generated in the test tool interface, which can facilitate the tester to intuitively see the results of the UI compatibility test.

[0098] In some examples, the type of abnormal annotation can be a box selection annotation, that is, an abnormal UI element shown by an abnormal node is selected by using a striking rectangular box in the test tool interface. For details, the effect of the box selection annotation can be seen in Fig.10 shown.

[0099] Combination Figure 2-Figure 5 The method shown, the embodiment of the present application obtains UI hierarchical structure information of multiple devices, and then uses the nodes shown in the UI hierarchical structure information and the corresponding attributes to determine the abnormal nodes in the device to be tested, thereby realizing effective detection of UI compatibility, and also generates corresponding abnormal annotations in the test tool interface according to the display position of the abnormal nodes, so as to facilitate testers to intuitively see the results of UI compatibility detection.

[0100] The process shown in S101-S106 above uses the UI hierarchical structure information of multiple devices to identify abnormal nodes, which can eliminate the detection errors caused by different types and resolutions among multiple devices in the prior art, thereby effectively improving the accuracy of UI compatibility detection.

[0101] like Figure 2 As shown, it is a flow chart of another UI compatibility detection method provided in an embodiment of the present application, including the following steps.

[0102] S201: Determine a first node set based on first node information.

[0103] The first node set includes multiple first nodes and corresponding attributes.

[0104] S202: Determine a second node set based on the second node information.

[0105] The second node set includes multiple second nodes and corresponding attributes.

[0106] S203: Matching the plurality of first nodes and the plurality of second nodes according to corresponding attributes to obtain matched points and unmatched points.

[0107] The matched points represent the second nodes that are successfully matched, and the unmatched points represent the second nodes that are unmatched.

[0108] Optionally, if the attribute of the first node is the same as the attribute of the second node, it is determined that the first node matches the second node; if the attribute of the first node is different from the attribute of the second node, it is determined that the first node does not match the second node.

[0109] In some examples, the resolutions of the device under test and the reference device may be different, that is, the bounds value of the root node corresponding to the device under test is different from the bounds value of the root node corresponding to the reference device. Therefore, the different attributes of the root nodes between the device under test and the reference device can be ignored, that is, the root node of the device under test and the root node of the reference device are considered to match.

[0110] In some examples, the ordering of the sibling nodes of the device under test and the reference device may be different. For example, the index value of the first node corresponding to the reference device is 1, and the index value of the second node corresponding to the device under test is 0. However, the first node and the second node are both nodes corresponding to the same UI element in the page image under test. For this reason, the difference in the index of the nodes between the device under test and the reference device can be ignored, that is, the first node and the second node with the same parameters in the attributes except the index are regarded as matching.

[0111] S204: Determine whether the attribute of the matched point is abnormal according to the attribute of the first node that matches the matched point.

[0112] Among them, after determining the matched points, if the resolutions of the device to be tested and the reference device are different (resulting in differences in the image coordinate systems of the device to be tested and the reference device), the node coordinates of the matched first node can be converted from the image coordinate system of the reference device to the image coordinate system of the device to be tested based on the attributes of the first node that matches the matched points (such as node coordinates) to obtain the corresponding estimated node coordinates (the coordinates can be regarded as the correct position of the corresponding UI element), and the estimated node coordinates are compared with the node coordinates of the matched points to determine whether the attributes of the matched points are abnormal.

[0113] Optionally, the implementation process of determining whether the attribute of the matched point is abnormal according to the attribute of the first node that matches the matched point can be found in Figure 3 The steps are shown and the corresponding explanations.

[0114] S205: Determine matched points and unmatched points with abnormal attributes as abnormal nodes.

[0115] Among them, the abnormal node determined based on the matched points and unmatched points of the attribute anomaly, and the abnormal UI element corresponding to the abnormal node can be understood as the display effect of the abnormal UI element on the device under test, which is different from its own display effect on the reference device.

[0116] In the process shown in S201-S205 above, the first node and the second node are matched according to the corresponding attributes to obtain matched points and unmatched points, and then the attributes of the first node that matches the matched points are used to determine whether the attributes of the matched points are abnormal, thereby obtaining matched points with abnormal attributes, thereby achieving effective identification of abnormal nodes and improving the accuracy of UI compatibility detection.

[0117] like Figure 3 As shown, it is a flow chart of another UI compatibility detection method provided in an embodiment of the present application, including the following steps.

[0118] S301: Obtain first picture size information and second picture size information.

[0119] The first screen size information is the screen size information of the reference device, and the second screen size information is the screen size information of the device to be tested.

[0120] It should be noted that the method for obtaining the screen size information of the reference device and the device under test can refer to the explanation of the step shown in the above S101.

[0121] In some examples, if the reference device and the device under test have different device models and / or different resolutions, the first screen size information is different from the second screen size information.

[0122] S302: Determine estimated node coordinates based on the first picture size information, the second picture size information, and an attribute of a first node that matches the matched point.

[0123] The implementation process of determining the estimated node coordinates based on the first screen size information, the second screen size information, and the attributes of the first node that matches the matched point can be found in Figure 4 The steps are shown and the corresponding explanations.

[0124] S303: Determine whether the attributes of the matched points are abnormal based on the estimated node coordinates.

[0125] Among them, based on the attributes of the matched points, the node coordinates of the matched points can be obtained, and the node coordinates of the matched points can be compared with the estimated node coordinates to determine whether the attributes of the matched points are abnormal.

[0126] In some examples, if the node coordinates of the matched point are the same as the estimated node coordinates, the attribute of the matched point can be determined to be normal, and if the node coordinates of the matched point are different from the estimated node coordinates, the attribute of the matched point can be determined to be abnormal.

[0127] It should be noted that in the process of comparing the node coordinates of the matched points with the estimated node coordinates, slight errors at the pixel level are difficult to distinguish with the naked eye. In order to eliminate slight errors, a corresponding error threshold can be set. If the error between the node coordinates of the matched points and the estimated node coordinates meets the error threshold, the attributes of the matched points are determined to be normal. If the error between the node coordinates of the matched points and the estimated node coordinates does not meet the error threshold, the attributes of the matched points are determined to be abnormal.

[0128] Optionally, the implementation process of determining whether the attributes of the matched points are abnormal based on the estimated node coordinates can be: based on the attributes of the matched points, obtaining the node coordinates of the matched points; determining the error between the estimated node coordinates and the node coordinates of the matched points; if the error meets the preset requirements, determining that the attributes of the matched points are normal; if the error does not meet the preset requirements, determining that the attributes of the matched points are abnormal.

[0129] In some examples, the preset requirement may be set as: the error satisfies an error threshold (eg, 10).

[0130] Specifically, assuming that the estimated node coordinates are [276, 1422][392, 1504], and the node coordinates of the matched points are [278, 1423][390, 1504], |278-276|<10, |1422-1423|<10, |392-390|<10, |1504-1504|<10, then the attributes of the matched points are determined to be normal.

[0131] Specifically, assuming that the estimated node coordinates are [276, 1422][392, 1524], and the node coordinates of the matched points are [278, 1423][390, 1504], |278-276|<10, |1422-1423|<10, |392-390|<10, |1524-1504|>10, then the attributes of the matched points are determined to be abnormal.

[0132] The process shown in the above S301-S303 uses the attributes of the first node that matches the matched point to determine the corresponding estimated node coordinates, and determines whether the attributes of the matched point are abnormal based on the estimated node coordinates, thereby achieving effective identification of abnormal nodes and improving the accuracy of UI compatibility detection.

[0133] like Figure 4 As shown, it is a flow chart of another UI compatibility detection method provided in an embodiment of the present application, including the following steps.

[0134] S401: Obtain a first width value and a first height value based on first picture size information.

[0135] The first width value may be regarded as the width of the output screen of the reference device, and the first height value may be regarded as the height of the output screen of the reference device.

[0136] S402: Obtain a second width value and a second height value based on the second picture size information.

[0137] The second width value may be regarded as the width of the screen output by the device under test, and the second height value may be regarded as the height of the screen output by the device under test.

[0138] S403: Based on the attribute of the first node that matches the matched point, obtain the node coordinates [x1, y1] [x2, y2] of the first node that matches the matched point.

[0139] Among them, the attributes include a node coordinate system, and in the UI hierarchical structure information, the expression of the node coordinate is usually recorded in the format of [x, y][x', y'], where [x, y] represents the upper left corner coordinate, and [x', y'] represents the lower right corner coordinate.

[0140] S404: Based on the first width value, the first height value, the second width value, the second height value, and the node coordinates [x1, y1] [x2, y2], the estimated node coordinates [x3, y3] [x4, y4] are calculated.

[0141] Among them, x3=x1×(second width value / first width value), y3=y1×(second height value / first height value), x4=x2×(second width value / first width value), y4=y2×(second height value / first height value).

[0142] In some examples, taking the scene of matched points corresponding to UI elements (such as buttons) as an example, the first width value is recorded as 1920, the first height value is recorded as 1080, the second width value is recorded as 3840, and the second height value is recorded as 2160. Device 1 represents the reference device, and device 2 represents the device to be tested. The node coordinates of the first node matching the matched points are [138, 711][196, 752]. The calculated estimated node coordinates are [x3, y3][x4, y4]. Specifically, x3=138×(3840 / 1920)=276, y3=711×(2160 / 1080)=1422, x4=196×(3840 / 1920)=392, y4=752×(2160 / 1080)=1504, that is, the estimated node coordinates are [276, 1422][392, 1504].

[0143] The process shown in S401-S404 above can use the first picture size information and the second picture size information to transform the node coordinates of the first node matching the matched point from the image coordinate system of the reference device to the image coordinate system of the device under test to obtain the corresponding estimated node coordinates.

[0144] like Figure 5 As shown, it is a flow chart of another UI compatibility detection method provided in an embodiment of the present application, including the following steps.

[0145] S501: Obtain a second width value and a second height value based on the screen size information of the device to be tested.

[0146] The second width value may be regarded as the width of the screen output by the device under test, and the second height value may be regarded as the height of the screen output by the device under test.

[0147] S502: Obtain a third width value and a third height value based on the image size information of the page image to be tested.

[0148] The third width value may be regarded as the image width of the page image to be tested, and the third height value may be regarded as the image height of the page image to be tested.

[0149] It should be noted that, the image size information of the page images to be tested output by the multiple devices displayed on the test tool interface is unified.

[0150] S503: Obtain a fourth width value and a fourth height value based on the interface size information of the test tool interface.

[0151] Among them, the fourth width value can be regarded as the display width of the test tool interface, and the fourth height value can be regarded as the display height of the test tool interface.

[0152] S504: Based on the attributes of the abnormal node, obtain the node coordinates [x5, y5] [x6, y6] of the abnormal node.

[0153] Among them, x5, y5, x6, and y6 can be natural numbers.

[0154] S505: Based on the second width value, the second height value, the third width value, the third height value, the fourth width value, the fourth height value, and the node coordinates [x5, y5][x6, y6], the target node coordinates [x7, y7][x8, y8] are calculated.

[0155] Among them, x7=x5×(third width value / second width value)+(fourth width value-third width value) / 2, y7=y5×(third height value / second height value)+(fourth height value-third height value) / 2, x8=x6×(third width value / second width value)+(fourth width value-third width value) / 2, y8=y6×(third height value / second height value)+(fourth height value-third height value) / 2.

[0156] S506: Based on the target node coordinates [x7, y7] [x8, y8], determine the display position of the abnormal node in the test tool interface.

[0157] Among them, based on the target node coordinates [x7, y7][x8, y8], the display position of the abnormal UI element shown by the abnormal node in the test tool interface can be determined.

[0158] The process shown in S501-S506 above can use the screen size information of the device to be tested, the image size information of the page image to be tested, and the interface size information of the test tool interface to convert the node coordinates of the abnormal node into the target node coordinate system to obtain the display position of the abnormal node in the test tool interface.

[0159] like Figure 6 As shown, it is a schematic diagram of the architecture of a UI compatibility detection device provided in an embodiment of the present application, including the units shown below.

[0160] The structure information collecting unit 100 is used to obtain UI hierarchical structure information when multiple devices play the same page image to be tested through the test tool interface; the multiple devices include a reference device and a device to be tested.

[0161] The first node analysis unit 200 is used to obtain corresponding first node information based on UI hierarchical structure information of a reference device.

[0162] The second node analysis unit 300 is used to obtain corresponding second node information based on the UI hierarchical structure information of the device under test.

[0163] The abnormal node identification unit 400 is used to determine an abnormal node based on the first node information and the second node information; the abnormal node is used to represent an abnormal UI element that is different between the device under test and the reference device when playing the same page image under test.

[0164] Optionally, the abnormal node identification unit 400 is specifically used to: determine a first node set based on first node information; the first node set includes multiple first nodes and corresponding attributes; determine a second node set based on second node information; the second node set includes multiple second nodes and corresponding attributes; match multiple first nodes and multiple second nodes according to corresponding attributes to obtain matched points and unmatched points; the matched points represent second nodes that have successfully matched, and the unmatched points represent second nodes that have failed to match; determine whether the attributes of the matched points are abnormal based on the attributes of the first nodes that match the matched points; and determine the matched points and unmatched points with abnormal attributes as abnormal nodes.

[0165] Optionally, the abnormal node identification unit 400 is specifically used to: if the attribute of the first node is the same as the attribute of the second node, determine that the first node matches the second node; if the attribute of the first node is different from the attribute of the second node, determine that the first node does not match the second node.

[0166] Optionally, the abnormal node identification unit 400 is specifically used to: obtain first screen size information and second screen size information; the first screen size information is the screen size information of the reference device, and the second screen size information is the screen size information of the device to be tested; based on the first screen size information, the second screen size information, and the attributes of the first node that matches the matched point, determine the estimated node coordinates; based on the estimated node coordinates, determine whether the attributes of the matched point are abnormal.

[0167] Optionally, the abnormal node identification unit 400 is specifically used to: obtain a first width value and a first height value based on the first screen size information; obtain a second width value and a second height value based on the second screen size information; obtain node coordinates [x1, y1][x2, y2] of the first node that matches the matched point based on the attribute of the first node that matches the matched point; calculate estimated node coordinates [x3, y3][x4, y4] based on the first width value, the first height value, the second width value, the second height value, and the node coordinates [x1, y1][x2, y2]; wherein, x3=x1×(second width value / first width value), y3=y1×(second height value / first height value), x4=x2×(second width value / first width value), y4=y2×(second height value / first height value).

[0168] Optionally, the abnormal node identification unit 400 is specifically used to: obtain the node coordinates of the matched point based on the attributes of the matched point; determine the error between the estimated node coordinates and the node coordinates of the matched point; if the error meets the preset requirements, determine that the attributes of the matched point are normal; if the error does not meet the preset requirements, determine that the attributes of the matched point are abnormal.

[0169] The abnormal position identification unit 500 is used to determine the display position of the abnormal node in the test tool interface based on the screen size information of the device to be tested, the image size information of the page image to be tested, and the interface size information of the test tool interface.

[0170] Optionally, the abnormal position identification unit 500 is specifically used to: obtain a second width value and a second height value based on the screen size information of the device to be tested; obtain a third width value and a third height value based on the image size information of the page image to be tested; obtain a fourth width value and a fourth height value based on the interface size information of the test tool interface; obtain the node coordinates [x5, y5][x6, y6] of the abnormal node based on the attributes of the abnormal node; calculate the target node based on the second width value, the second height value, the third width value, the third height value, the fourth width value, the fourth height value, and the node coordinates [x5, y5][x6, y6]. Node coordinates [x7, y7][x8, y8]; wherein, x7=x5×(third width value / second width value)+(fourth width value-third width value) / 2, y7=y5×(third height value / second height value)+(fourth height value-third height value) / 2, x8=x6×(third width value / second width value)+(fourth width value-third width value) / 2, y8=y6×(third height value / second height value)+(fourth height value-third height value) / 2; based on the target node coordinates [x7, y7][x8, y8], determine the display position of the abnormal node in the test tool interface.

[0171] The abnormal annotation generating unit 600 is used to generate a corresponding abnormal annotation in the test tool interface according to the display position of the abnormal node.

[0172] Each of the above-mentioned units utilizes the UI hierarchical structure information of multiple devices to identify abnormal nodes, and can eliminate detection errors caused by different types and resolutions among multiple devices in the prior art, thereby effectively improving the accuracy of UI compatibility detection.

[0173] The present application also provides a computer-readable storage medium, which includes a stored program, wherein the program executes the UI compatibility detection method provided by the present application.

[0174] The present application also provides an electronic device, including: a processor, a memory and a bus. The processor and the memory are connected via the bus, the memory is used to store programs, and the processor is used to run the programs, wherein the UI compatibility detection method provided by the present application is executed when the programs are run.

[0175] In addition, the functions described above in the embodiments of the present application may be at least partially performed by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0176] Although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present application. Certain features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0177] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A UI compatibility detection method, characterized in that: include: Obtain the UI hierarchical structure information when multiple devices play the same page image to be tested through the test tool interface; The multiple devices include a reference device and a device under test; Based on the UI hierarchical structure information of the reference device, obtaining corresponding first node information; Based on the UI hierarchical structure information of the device under test, obtaining corresponding second node information; Determine an abnormal node based on the first node information and the second node information; The abnormal node is used to represent an abnormal UI element that is different between the device under test and the reference device when playing the same page image under test; Determine the display position of the abnormal node in the test tool interface based on the screen size information of the device to be tested, the image size information of the page image to be tested, and the interface size information of the test tool interface; According to the display position of the abnormal node, a corresponding abnormal annotation is generated in the test tool interface.

2. The method according to claim 1, characterized in that Determining an abnormal node based on the first node information and the second node information includes: Based on the first node information, determining a first node set; the first node set includes a plurality of first nodes and corresponding attributes; Based on the second node information, determining a second node set; the second node set includes a plurality of second nodes and corresponding attributes; According to corresponding attributes, a plurality of the first nodes and a plurality of the second nodes are matched to obtain matched points and unmatched points; the matched points represent second nodes that are successfully matched, and the unmatched points represent second nodes that fail to match; Determining whether the attribute of the matched point is abnormal according to the attribute of the first node matched with the matched point; The matched points with abnormal attributes and the unmatched points are determined as abnormal nodes.

3. The method according to claim 2, characterized in that If the attribute of the first node is the same as the attribute of the second node, it is determined that the first node matches the second node; if the attribute of the first node is different from the attribute of the second node, it is determined that the first node does not match the second node.

4. The method according to claim 2, characterized in that: Determining whether the attribute of the matched point is abnormal according to the attribute of the first node matched with the matched point includes: Obtaining first screen size information and second screen size information; the first screen size information is the screen size information of the reference device, and the second screen size information is the screen size information of the device under test; Determine estimated node coordinates based on the first picture size information, the second picture size information, and an attribute of a first node that matches the matched point; According to the estimated node coordinates, it is determined whether the attributes of the matched points are abnormal.

5. The method according to claim 4, characterized in that Determining estimated node coordinates based on the first picture size information, the second picture size information, and an attribute of a first node that matches the matched point includes: Based on the first picture size information, obtaining a first width value and a first height value; Based on the second picture size information, obtaining a second width value and a second height value; Based on the attribute of the first node that matches the matched point, obtain the node coordinates [x1, y1] [x2, y2] of the first node that matches the matched point; Based on the first width value, the first height value, the second width value, the second height value, and the node coordinates [x1, y1][x2, y2], the estimated node coordinates [x3, y3][x4, y4] are calculated; wherein, x3=x1×(second width value / first width value), y3=y1×(second height value / first height value), x4=x2×(second width value / first width value), y4=y2×(second height value / first height value).

6. The method according to claim 4, characterized in that Determining whether the attribute of the matched point is abnormal according to the estimated node coordinates includes: Based on the attributes of the matched points, obtaining the node coordinates of the matched points; Determining the error between the estimated node coordinates and the node coordinates of the matched points; If the error meets the preset requirement, determining that the attribute of the matched point is normal; If the error does not meet the preset requirement, it is determined that the attribute of the matched point is abnormal.

7. The method according to claim 1, characterized in that Determining a display position of the abnormal node in the test tool interface based on the screen size information of the device to be tested, the image size information of the page image to be tested, and the interface size information of the test tool interface includes: Based on the screen size information of the device under test, obtaining a second width value and a second height value; Based on the image size information of the page image to be tested, obtaining a third width value and a third height value; Based on the interface size information of the test tool interface, obtaining a fourth width value and a fourth height value; Based on the attributes of the abnormal node, obtain the node coordinates [x5, y5] [x6, y6] of the abnormal node; Based on the second width value, the second height value, the third width value, the third height value, the fourth width value, the fourth height value, and the node coordinates [x5, y5][x6, y6], the target node coordinates [x7, y7][x8, y8] are calculated; wherein, x7=x5×(third width value / second width value)+(fourth width value-third width value) / 2, y7=y5×(third height value / second height value)+(fourth height value-third height value) / 2, x8=x6×(third width value / second width value)+(fourth width value-third width value) / 2, y8=y6×(third height value / second height value)+(fourth height value-third height value) / 2; Based on the target node coordinates [x7, y7] [x8, y8], the display position of the abnormal node in the test tool interface is determined.

8. A UI compatibility detection device, characterized in that: include: A structural information collection unit, used to obtain UI hierarchical structure information when multiple devices play the same page image to be tested through the test tool interface; The multiple devices include a reference device and a device under test; A first node analysis unit, configured to obtain corresponding first node information based on UI hierarchical structure information of the reference device; A second node analysis unit, configured to obtain corresponding second node information based on the UI hierarchical structure information of the device under test; an abnormal node identification unit, configured to determine an abnormal node based on the first node information and the second node information; The abnormal node is used to represent an abnormal UI element that is different between the device under test and the reference device when playing the same page image under test; an abnormal position identification unit, configured to determine a display position of the abnormal node in the test tool interface based on the screen size information of the device to be tested, the image size information of the page image to be tested, and the interface size information of the test tool interface; The abnormal annotation generating unit is used to generate a corresponding abnormal annotation in the test tool interface according to the display position of the abnormal node.

9. A storage medium, characterized in that: The storage medium includes a stored program, wherein the program, when executed by a processor, executes the UI compatibility detection method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: processor, memory, and bus; The processor is connected to the memory via the bus; The memory is used to store programs, and the processor is used to run programs, wherein the program, when run by the processor, executes the UI compatibility detection method according to any one of claims 1 to 7.

Images