Automated testing for checking user interface truncation

Abstract

An information handling system detects whether a text element in a graphical user interface is truncated by comparing the text element to an expected text element using a pattern matching algorithm. The system also detects whether a non-textual element in the graphical user interface is truncated by comparing the non-textual element to a reconstructed copy of the non-textual element.

Description

Technical Field

[0001] The present disclosure relates generally to information handling systems and, more particularly, to automated testing for checking for user interface truncation. Background Art

[0002] As the value and use of information continue to grow, individuals and businesses seek additional ways to process and store information. One option is an information handling system. An information handling system typically processes, compiles, stores or transmits information or data for commercial, personal or other purposes. Technology and information handling needs and requirements may vary between different applications. Therefore, information handling systems may also differ in terms of what information is handled, how the information is handled, how much information is processed, stored or transmitted, and how quickly and efficiently the information can be processed, stored or transmitted. The changes in information handling systems allow information handling systems to be general or configured for specific users or specific purposes (such as financial transaction processing, airline ticketing, enterprise data storage or global communications). In addition, an information handling system may include a variety of hardware and software resources that can be configured to process, store and transmit information, and may include one or more computer systems, graphical interface systems, data storage systems, networking systems and mobile communication systems. Information handling systems can also implement various virtualization architectures. Data and voice communications in information handling systems can be carried out via a network, which is wired, wireless or a combination thereof. Summary of the invention

[0003] An information handling system detects whether a text element in a graphical user interface is truncated by comparing the text element with an expected text element using a pattern matching algorithm. The system also detects whether a non-text element in the graphical user interface is truncated by comparing the non-text element with a reconstructed copy of the non-text element. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] It should be understood that for simplicity and clarity of illustration, the elements shown in the figures are not necessarily drawn to scale. For example, the dimensions of some elements may be exaggerated relative to other elements. Embodiments incorporating the teachings of the present disclosure are shown and described with respect to the drawings herein, in which:

[0005] Figure 1 is a block diagram illustrating an information handling system according to an embodiment of the present disclosure;

[0006] Figure 2 is a block diagram of a graphical user interface (GUI) testing system for automatically checking for truncation of GUI elements according to an embodiment of the present disclosure;

[0007] Figure 3is a flow chart of a method for automatically checking GUI element truncation according to an embodiment of the present disclosure;

[0008] Figure 4 is a diagram of a GUI having identified text elements and non-text elements after automatic layout analysis according to an embodiment of the present disclosure;

[0009] Figure 5 is a diagram of a GUI image with identified text regions and non-text regions after automatic layout analysis according to an embodiment of the present disclosure;

[0010] Figure 6 and Figure 7 is a diagram of a GUI image with text that is dynamically populated with different content at runtime after automatic layout analysis according to an embodiment of the present disclosure; and

[0011] Figure 8 and Fig. 9 is a diagram of a non-text region having truncation issues according to an embodiment of the present disclosure.

[0012] The use of the same reference numbers in different drawings indicates similar or identical items. DETAILED DESCRIPTION

[0013] The following description in conjunction with the accompanying drawings is provided to help understand the teaching disclosed herein. The description focuses on the specific implementation and implementation of the teaching and is provided to help describe the teaching. This focus should not be interpreted as limiting the scope or applicability of the teaching.

[0014] Figure 1An embodiment of an information handling system 100 is shown, which includes processors 102 and 104, a chipset 110, a memory 120, a graphics adapter 130 connected to a video display 134, a non-volatile RAM (NV-RAM) 140 including a basic input and output system / extensible firmware interface (BIOS / EFI) module 142, a disk controller 150, a hard disk drive (HDD) 154, an optical drive 156, a disk emulator 160 connected to a solid state drive (SSD) 164, an input / output (I / O) interface 170 connected to extended resources 174 and a trusted platform module (TPM) 176, a network interface 180, and a baseboard management controller (BMC) 190. Processor 102 is connected to chipset 110 via processor interface 106, and processor 104 is connected to chipset via processor interface 108. In a specific embodiment, processors 102 and 104 are connected together via a high-capacity coherent fabric such as a HyperTransport link, a QuickPath Interconnect, or the like. Chipset 110 represents an integrated circuit or a group of integrated circuits that manages the flow of data between processors 102 and 104 and other elements of information handling system 100. In a particular embodiment, chipset 110 represents a pair of integrated circuits, such as a northbridge component and a southbridge component. In another embodiment, some or all of the functions and features of chipset 110 are integrated with one or more of processors 102 and 104.

[0015] Memory 120 is connected to chipset 110 via memory interface 122. Examples of memory interface 122 include double data rate (DDR) memory channels, and memory 120 represents one or more DDR dual in-line memory modules (DIMMs). In a specific embodiment, memory interface 122 represents two or more DDR channels. In another embodiment, one or more of processors 102 and 104 include a memory interface that provides dedicated memory for the processor. DDR channels and connected DDR DIMMs can comply with specific DDR standards, such as DDR3 standards, DDR4 standards, DDR5 standards, etc.

[0016] The memory 120 may also represent various combinations of memory types, such as dynamic random access memory (DRAM) DIMMs, static random access memory (SRAM) DIMMs, non-volatile DIMMs (NV-DIMMs), storage class memory devices, read-only memory (ROM) devices, etc. The graphics adapter 130 is connected to the chipset 110 via a graphics interface 132 and provides a video display output 136 to a video display 134. As needed or desired, examples of the graphics interface 132 include a peripheral component express interconnect (PCIe) interface, and the graphics adapter 130 may include a four-lane (x4) PCIe adapter, an eight-lane (x8) PCIe adapter, a 16-lane (x16) PCIe adapter, or another configuration. In a specific embodiment, the graphics adapter 130 is set down on a system printed circuit board (PCB). The video display output 136 may include a digital video interface (DVI), a high-definition multimedia interface (HDMI), a DisplayPort interface, etc., and the video display 134 may include a monitor, a smart TV, an embedded display such as a laptop computer display, etc.

[0017] NV-RAM 140, disk controller 150, and I / O interface 170 are connected to chipset 110 via I / O channels 112. Examples of I / O channels 112 include one or more point-to-point PCIe links between chipset 110 and each of NV-RAM 140, disk controller 150, and I / O interface 170. Chipset 110 may also include one or more other I / O interfaces, including a PCIe interface, an Industry Standard Architecture (ISA) interface, a Small Computer Serial Interface (SCSI) interface, an Inter-Integrated Circuit (IIC) interface, and a SCSI interface. 2 NV-RAM 140 includes a BIOS / EFI module 142 that stores machine executable code (BIOS / EFI code) that operates to detect resources of information handling system 100, provide drivers for the resources, initialize the resources, and provide common access mechanisms for the resources. The functions and features of BIOS / EFI module 142 will be further described below.

[0018] The disk controller 150 includes a disk interface 152 that connects the disk controller to a hard disk drive (HDD) 154, an optical disk drive (ODD) 156, and a disk emulator 160. Examples of the disk interface 152 include an integrated drive electronics (IDE) interface, an advanced technology attachment (ATA) such as a parallel ATA (PATA) interface or a serial ATA (SATA) interface, a SCSI interface, a USB interface, a proprietary interface, or a combination thereof. The disk emulator 160 allows the SSD 164 to be connected to the information handling system 100 via the external interface 162. Examples of the external interface 162 include a USB interface, an Institute of Electrical and Electronics Engineers (IEEE) 1394 (Firewire) interface, a proprietary interface, or a combination thereof. Alternatively, the SSD 164 can be disposed within the information handling system 100.

[0019] The I / O interface 170 includes a peripheral interface 172 that connects the I / O interface to an extended resource 174, a TPM 176, and a network interface 180. The peripheral interface 172 may be an interface of the same type as the I / O channel 112, or may be an interface of a different type. Thus, when the peripheral interface 172 and the I / O channel 112 are of the same type, the I / O interface 170 extends the capabilities of the I / O channel, and when they are of different types, the I / O interface converts information from a format suitable for the I / O channel to a format suitable for the peripheral interface 172. The extended resource 174 may include a data storage system, an additional graphics interface, a network interface card (NIC), a sound / video processing card, another extended resource, or a combination thereof. The extended resource 174 may be located on a main circuit board, on a separate circuit board or expansion card disposed within the information handling system 100, on a device external to the information handling system, or a combination thereof.

[0020] The network interface 180 represents a network communication device that is disposed within the information handling system 100, located on the main circuit board of the information handling system, integrated into another component such as the chipset 110, located in another suitable location, or a combination thereof. The network interface 180 includes a network channel 182 that provides an interface for devices external to the information handling system 100. In certain embodiments, the network channel 182 is a different type than the peripheral interface 172, and the network interface 180 converts information from a format suitable for the peripheral channel to a format suitable for the external device.

[0021] In certain embodiments, network interface 180 includes a NIC or a host bus adapter (HBA), and examples of network channel 182 include an InfiniBand channel, a Fibre Channel, a Gigabit Ethernet channel, a proprietary channel architecture, or a combination thereof. In another embodiment, network interface 180 includes a wireless communication interface, and network channel 182 includes a Wi-Fi channel, a Near Field Communication (NFC) channel, a The network channel 182 may be connected to an external network resource (not shown). The network resource may include another information handling system, a data storage system, another network, a grid management system, another suitable resource, or a combination thereof.

[0022] The BMC 190 is connected to multiple elements of the information handling system 100 via one or more management interfaces 192 to provide out-of-band monitoring, maintenance, and control of the elements of the information handling system. Therefore, the BMC 190 represents a processing device different from the processor 102 and the processor 104, which provides various management functions for the information handling system 100. For example, the BMC 190 may be responsible for power management, cooling management, etc. The term BMC is generally used in the context of server systems, while in consumer-grade devices, the BMC may be referred to as an embedded controller (EC). The BMC included in the data storage system may be referred to as a storage chassis processor. The BMC included in the chassis of a blade server may be referred to as a chassis management controller, and the embedded controller included at the blade of the blade server may be referred to as a blade management controller. The capabilities and functions provided by the BMC 190 may vary greatly based on the type of information handling system. The BMC 190 may operate according to the Intelligent Platform Management Interface (IPMI). Examples of BMC 190 include integrated Remote Access Controller (iDRAC).

[0023] Management interface 192 represents one or more out-of-band communication interfaces between BMC 190 and elements of information handling system 100, and may include I 2C bus, system management bus (SMBus), power management bus (PMBUS), low pin count (LPC) interface, serial bus such as universal serial bus (USB) or serial peripheral interface (SPI), network interface such as Ethernet interface, high-speed serial data link such as PCIe interface, network controller sideband interface (NC-SI), etc. As used herein, out-of-band access refers to operations performed on the information handling system 100 separately from the BIOS / operating system execution environment (i.e., separately from the execution of code by processors 102 and 104 and the programs implemented on the information handling system in response to the executed code).

[0024] As needed or desired, BMC 190 operates to monitor and maintain system firmware, such as code stored in BIOS / EFI module 142, option ROM for graphics adapter 130, disk controller 150, expansion resources 174, network interface 180, or other elements of information handling system 100. Specifically, BMC 190 includes a network interface 194, which can be connected to a remote management system to receive firmware updates as needed or desired. Here, BMC 190 receives the firmware update, stores the update to a data storage device associated with the BMC, transmits the firmware update to the NV-RAM of the device or system that is the target of the firmware update, thereby replacing the currently operating firmware associated with the device or system, and restarts the information handling system, which then utilizes the updated firmware image.

[0025] BMC 190 utilizes various protocols and application programming interfaces (APIs) to direct and control the process for monitoring and maintaining system firmware. Examples of protocols or APIs for monitoring and maintaining system firmware include a graphical user interface (GUI) associated with BMC 190, an interface defined by the Distributed Management Task Force (DMTF) such as the Web Services Management (WSMan) interface, the Management Component Transport Protocol (MCTP), or the like, as needed or desired. interfaces), various vendor-defined interfaces (such as the Dell EMC Remote Access Controller Manager (RACADM) utility, Dell EMC OpenManage Enterprise, Dell EMC OpenManage Server Administrator (OMSA) utility, Dell EMC OpenManage Storage Services (OMSS) utility, or the Dell EMC OpenManage Deployment Toolkit (DTK) kit), a BIOS setup utility such as invoked by the "F2" boot option, or another protocol or API.

[0026] In a particular embodiment, BMC 190 is included on a main circuit board (such as a substrate, a motherboard, or any combination thereof) of information handling system 100, or is integrated into another element (such as chipset 110) or another suitable element of the information handling system, as needed or desired. Therefore, BMC 190 can be part of an integrated circuit or chipset within information handling system 100. Examples of BMC 190 include iDRAC, etc. BMC 190 can operate on a power plane separate from other resources in information handling system 100. Therefore, when the resources of information handling system 100 are powered off, BMC 190 can communicate with the management system via network interface 194. Information can be sent from the management system to BMC 190, and the information can be stored in RAM or NV-RAM associated with BMC. Information stored in RAM may be lost after the power plane of BMC 190 is powered off, while information stored in NV-RAM can be saved by power-off / power-on cycles of the power plane of BMC.

[0027] The information handling system 100 may include additional components and additional buses that are not shown for clarity. For example, the information handling system 100 may include multiple processor cores, audio devices, etc. Although a specific arrangement of bus technologies and interconnections is shown for illustrative purposes, those skilled in the art will appreciate that the techniques disclosed herein may be applicable to other system architectures. The information handling system 100 may include multiple central processing units (CPUs) and redundant bus controllers. One or more components may be integrated together. The information handling system 100 may include additional buses and bus protocols, such as I 2 C, etc. Additional components of information handling system 100 may include one or more storage devices that can store machine-executable code, one or more communication ports for communicating with external devices, and various input and output (I / O) devices (such as a keyboard, mouse, and video display).

[0028] For purposes of this disclosure, information handling system 100 may include any tool or collection of tools operable to compute, classify, process, transmit, receive, retrieve, generate, switch, store, display, indicate, detect, record, reproduce, dispose of, or utilize any form of information, intelligence, or data for commercial, scientific, control, entertainment, or other purposes. For example, information handling system 100 may be a personal computer, a laptop computer, a smart phone, a tablet device or other consumer electronic device, a network server, a network storage device, a switch, a router or another network communication device, or any other suitable device, and may vary in size, shape, performance, functionality, and price. In addition, information handling system 100 may include processing resources for executing machine executable code, such as processor 102, a programmable logic array (PLA), an embedded device such as a system on a chip (SoC), or other control logic hardware. Information handling system 100 may also include one or more computer-readable media for storing machine executable code such as software or data.

[0029] A typical graphical user interface (GUI) includes elements such as text, input controls, navigation components, and information components. Although GUI elements may look normal under a specific resolution, operating system, or browser, these elements may be accidentally truncated when displayed using different resolutions and / or localized into different languages. For example, text and GUI areas may be truncated or cut off. GUI testing includes detecting GUI elements and determining the truncation of GUI elements (if any). Typically, testers manually identify truncation issues by viewing the GUI layout during the testing process. However, for multiple languages, operating systems, browsers, and resolutions, manually identifying truncated GUI elements is tedious, time-consuming, and expensive. In order to address these and other problems, the present disclosure provides a system and method for automated GUI testing that can check the truncation of GUI elements.

[0030] Figure 2 A GUI testing system 200 for automatically checking for truncation of GUI elements is shown. The GUI testing system 200 (which may be implemented as Figure 1 The representative state transfer service in the information handling system 100 of the present invention includes an image processor 205, a GUI element detector 210, a report generator 230, a preprocessor 235, a test engine 240, and a reconstruction engine 245. The GUI element detector 210 includes a text element detector 220 and a non-text element detector 225. The GUI testing system 200 is an automated testing framework that can be configured to automatically check for truncation in GUI elements of an application and generate a report on its findings.

[0031] During testing, several screens or pages of an application may be provided to the GUI testing system 200. One of the screens or pages may be used as a base GUI image, where its GUI elements may be used as a basis for comparison with other screens or pages to detect truncation issues. The base GUI image may have been pre-verified as not having truncated GUI elements. The GUI testing system 200 may traverse other screens or pages of the application, which may have different resolutions and / or languages, to compare each screen or page with the base GUI image.

[0032] Image processor 205 can be configured to capture an image of GUI, which can be a screen shot of a screen or page of an application. In addition, image processor 205 can extract desired text elements from a basic GUI image, such as by using optical character recognition technology. Expected text elements can be stored in one of several formats, such as JavaScript object notation files or similar files. Expected text elements can be used as a baseline for comparing text elements identified by GUI element detector 210. Information associated with expected GUI elements can be transmitted to test engine 240. In addition, basic GUI images can be used as a pre-trained model by text element detector 220 to detect text elements in the GUI image to be tested. Each of the GUI images to be tested can have a resolution and / or language different from the GUI basic image.

[0033] The GUI image to be tested may be transmitted to a preprocessor 235 for preprocessing, which includes removing noise associated with the image content of the GUI image. The preprocessor 235 may also adjust the color of the GUI elements to a single color, such as gray, before transmitting to the GUI element detector 210. This may contribute to more accurate GUI element detection.

[0034] The GUI element detector 210 may be configured to identify or recognize GUI elements in the test GUI image. The GUI element detector 210 may be configured to determine the location, size, and category of the GUI element and provide related Figure 4 . In addition, GUI element detector 210 can generate regions representing potential layout blocks of GUI elements. Regions associated with text elements can have different colors from regions associated with non-text elements to facilitate distinguishing these regions.

[0035] The text element detector 220 may be configured to detect text elements. The non-text element detector 225 may be configured to detect non-text elements, such as control elements, icons, cursors, etc. The text element detector 220 may use a deep learning method, such as the Efficient Accurate Scene Text (EAST) detector, to use a basic GUI image as a model to identify text elements, including text within non-text element regions. The text element detector 220 may detect text elements in a test GUI image by learning GUI element features and their composition from one or more basic GUI images. The text element detector 220 may generate text regions representing potential text elements, such as Figure 5 In addition, the text element detector 220 may consider the text content and layout characteristics of the text element, such as height, width, category, etc.

[0036] The non-text element detector 225 can be configured to use top-down computer vision techniques to detect instances and classify non-text elements in the test GUI image. In addition, the non-text element detector 225 can consider the boundaries, shapes, textures, and layout characteristics of the non-text GUI elements. The non-text element detector 225 can generate non-text regions representing potential non-text GUI elements, such as Figure 5 Based on the non-text area provided by the non-text element detector 225, the reconstruction engine 245 can be configured to reconstruct the shape detected by the non-text element detector 225 so that the area associated with the non-text element is closed or not closed.

[0037] The test engine 240 can be configured to check whether the text elements in the test GUI image identified by the text element detector 220 are the same as the expected text elements in the base GUI image. The test engine 240 can use the outputs from the image processor 205 and the GUI element detector 210 as input. The output from the image processor 205 can be used as the expected text element and compared with the output from the text element detector 220 using a pattern matching algorithm to determine whether the text elements in the test GUI image are truncated. The test engine 240 can use a shape reconstruction algorithm to determine whether the non-text elements detected by the non-text element detector 225 are truncated. The test engine 240 can provide output associated with the comparison. The report generator 230 can be configured to generate a report based on the output of the test engine 240. The report can identify the truncated GUI elements. The report can also identify the positions of these GUI elements.

[0038] Those of ordinary skill in the art will understand that Figure 2The configuration, hardware and / or software components of the depicted GUI testing system 200 may vary. For example, the illustrative components within the GUI testing system 200 are not intended to be exhaustive, but are represented to highlight components that can be used to implement various aspects of the present disclosure. For example, other devices and / or components may be used in addition to or in place of the depicted devices / components. The depicted examples do not convey or imply any architectural or other limitations with respect to the currently described embodiments and / or overall disclosure. In the discussion of the figures, for the sake of continuity of the description, reference may also be made to components shown in other figures. Therefore, various aspects of the present disclosure may be applied to or suitable for use in many other contexts.

[0039] The components of the GUI testing system 200 may be implemented in hardware, software, firmware, or any combination thereof. Additionally or alternatively, the GUI testing system 200 may include, in addition to Figure 2 Various additional components other than those shown in FIG. 1 and 2 may be included in the GUI testing system 200. For clarity of description, connections between components may be omitted. In various embodiments, the GUI testing system 200 may not include Figure 2 Each of the components shown in . In addition, Figure 2 Some components shown as separate components in the figure may be integrated with other components in some embodiments instead. For example, in some embodiments, all or part of the functionality provided by the components shown may be provided by components integrated into one or more processors as a system on a chip instead.

[0040] Figure 3 A method 300 for performing automated GUI testing to detect truncation issues of GUI elements is shown. The method 300 may be performed by Figure 2 However, despite the fact that Figure 2 The GUI test system 200 of the present disclosure describes an embodiment of the present disclosure, but it should be appreciated that other systems can be used to perform the described method. Those skilled in the art will appreciate that the flowchart explains a typical example and can be extended to advanced applications or services in practice.

[0041] Method 300 usually starts at frame 305, and the test interface of wherein test framework receives several application GUI screens or pages for testing.One of GUI screen or page can be used as the basis or model for detecting the GUI elements in other GUI screens or pages.In one embodiment, application program can include resource file, such as .resx file, and it is the non-executable data that is usually deployed with application program.Resource file can include various forms of data, such as string and image associated with application program.For example, resource file can include expected text element.Resource file can be one of various formats, such as JavaScript object notation format, extensible markup language format or similar format.

[0042] The method may proceed to frame 310, where the test framework may capture images, such as screenshots of each of the GUI screens or pages being tested. Each of the screenshots may also be referred to as a GUI image. The method may proceed to frame 315, where the test framework may extract text elements from a resource file associated with a basic GUI image of the application. The basic GUI image may be predetermined by a user to not include truncated text elements and non-text elements. The extracted text may be used as expected text for comparison with the detected GUI elements in the GUI image being tested. The method may proceed to frame 320.

[0043] At block 320, a preprocessor may preprocess the GUI image to be tested for truncation issues using computer vision techniques. For example, the preprocessor may remove noise and convert different colors of GUI elements to gray. The method may then proceed to block 325, where a GUI element detector may use a deep learning scene text detector (such as an EAST detector) to identify text elements in the GUI image. The deep learning scene text detector may provide information associated with the identified text elements, such as height, width, position, etc., such as Figure 4 The output 420 of FIG. 4 shows. In addition, the GUI element detector can use computer vision techniques to detect non-text elements in the GUI image. Computer vision techniques can be used to detect contour key points of non-text areas, such as Figure 4 The method may proceed to block 330.

[0044] At frame 330, the test engine can compare the identified text element with the expected text element. In order to compare the text elements, a pattern matching algorithm can be used to compare the identified text with the expected text. For example, the method can compare the text content in the text area 505 and the output 420 of the text element 405. The test engine can also compare the position, height, width and / or other attributes of the text area with the expected text. For example, the test engine can compare the height and width of the text area 510 with the height and width in the output 420 of the text element 410.

[0045] The method may proceed to block 335, where the test engine may compare the shape of the identified or detected non-text element or region with the reconstructed non-text element or region. A shape reconstruction algorithm may be used to cut out the non-text element region based on the contour key points of the non-text element region from the GUI image and save it as a file similar to Figure 8 The original image is preprocessed by changing the color of the original image to gray before saving the copy. The reconstruction engine can then overlay the outline of the copy of the original image, generate a reconstruction, and save it as the reconstructed copy. The shape of the original image can be compared to the shape of the reconstructed copy to determine whether the shapes match or are identical, such as Fig. 9 The method may proceed to block 340.

[0046] At block 340, the test engine may determine whether the detected or identified text elements and non-text elements are truncated or cut off. If, based on the comparison at block 330, the detected or identified text elements are not identical to the expected text elements, the detected or identified text elements may be truncated or cut off.

[0047] The comparison of the non-text elements detected or identified at box 335 can return a metric, wherein the lower the metric value, the better the match. A threshold value can be used, such as if the metric value returned is equal to or lower than the threshold value, then the images can be considered to be the same. If the metric value returned is greater than the threshold value, then the images may not be considered to be the same. In one example, the threshold value can be set to 0.3, which can be adjusted by the tester. For each non-text element being evaluated, the threshold value may be different. If the detected or identified image does not match or is not the same as the reconstructed image, then the detected or identified image may be truncated or cut off. Because the information associated with the non-text elements is used, the shape reconstruction algorithm can be applicable to any GUI implementation.

[0048] Blocks 310 to 335 may be performed for each of the received graphical user interface screens or pages. After processing each of the GUI images, the method may proceed to block 345 where a report may be generated based on the comparison results. The report may be in one of several formats, such as a JavaScript object notation format or a similar format.

[0049] Figure 4 An example of a GUI image 400 with identified text elements and non-text elements after automatic layout analysis is shown. GUI image 400 includes text elements 405 and 410 and non-text elements 415. In this example, output 420 is associated with identified text elements 405, and output 425 is associated with non-text elements 415. Therefore, output 420 includes information associated with text element 405, such as its category, height, width, position and text content. In this example, text element 405 is assigned with category text. Height, width and position can be in pixels. Height and width refer to the height and width of text. Position shows the position of text element in screenshot, which is further identified by its coordinates (such as column minimum, row minimum, column maximum and row maximum (respectively column_min, row_min, column_max and row_max)). For example, column_min can refer to the leftmost column occupied by GUI element, and row_min can refer to the bottommost row. Thus, column_max may refer to the rightmost column occupied by a GUI element, and row_max may refer to the topmost row.

[0050] Similarly, output 425 includes information associated with non-text element 415, such as its category, height, width and position. In this example, non-text element 415 is assigned a category composition control. Height, width and position can be in pixels. Height and width refer to the height and width of text. The position shows the position of non-text element in the screenshot, which is further identified by column_min, row_min, column_max and row_max.

[0051] Figure 5 An example of a GUI image 500 is shown, which shows text areas and non-text areas identified after automatic layout analysis. The text areas are associated with text elements, respectively, and the non-text areas are associated with non-text elements, respectively. In this example, the GUI image 500 includes text areas 505 and 510 and non-text area 515. In this example, the text area 505 is associated with the text element 405 and is not truncated. The text area 510 is associated with the text element 410 and is truncated. The non-text area 515 is associated with the non-text element 415 and is truncated.

[0052] Figure 6 An example of a GUI image 600 with text that is dynamically populated with different content at runtime after automatic layout analysis is shown. GUI image 600 includes text elements 605 and 610. Figure 7 An example of a GUI image 700 similar to GUI image 600 is shown. GUI image 700 includes text areas 705 and 710 corresponding to text elements 605 and 610, respectively. In this example, both text areas 705 and 710 are truncated. Because the text detected in GUI image 700 changes dynamically, the GUI testing system can utilize regular expressions to match the detected text with the expected text shown in GUI image 600. Table 1 shows an example of a regular expression that can be used to compare the expected text with the detected text.

[0053] Table 1

[0054] Expected value Detected value model {} minutes remaining 3 minutes remaining re.search(r″[1-9]^minutes remaining″,D_v) null C:\temp\DelNSARemediation\log\Bootupinit.bg Path(D_v).is_dir()or Path(D_v).is_file {}％ 40％ re.search(r″[1-9]^%″,D_v)

[0055] In addition, in some cases, text may be displayed next to other GUI elements, such as the symbols shown in Table 2 below. These situations may challenge accurate text detection. Therefore, text element detector 220 can take these other GUI elements into account. Table 2 shows an example of how to handle the comparison between expected text and detected text when expected text includes symbols close to characters in the text area. When the detected text includes a part of symbols or other GUI elements close to the text, a false negative may be reported. This is because these symbols may be mistakenly detected as special characters. Therefore, when the detected text is compared with the expected text, they will be different. In order to solve the false negative, the test element detector can filter out the symbols from the negative test results and compare them again, which should produce a positive match.

[0056] Table 2 includes an expected value column, an expected appearance column, a detected value column, and a text area column with filters applied. In this example, the "expected value" is "WHAT", which appears next to the symbol "②". When a specific symbol like ② is adjacent to the expected value, the detected value may include a portion of the symbol. In this example, the detected value is shown as ")WHAT", which includes the side portion of the symbol ②. Therefore, false negatives may be reported in the test results. To solve this problem, the text element detector can apply a symbol filter and perform a comparison between the expected value and the detected value again. At this time, the detected text area (as shown in the dotted box in the text area column) matches the expected value and returns a positive match.

[0057] In another example, the expected value "BACKUP PROGRESS" appears next to the symbol "④". The detected value is depicted as "\U2463 BACKUP PROGRESS", where "\U2463" is the Unicode character of the symbol ④. This may show another false negative in the test results. To address this issue, the text element detector can filter out the Unicode character of the symbol and / or provide a symbol for the Unicode character "circled number four", as shown in the text area of ​​the second row. Afterwards, the test engine can perform the comparison again, which should produce a positive match.

[0058] Table 2

[0059]

[0060] Figure 8 A non-text region 805 is shown with truncation issues and associated contour keypoints. The non-text region 805 may be Figure 5 A copy or cut of the non-text area 515 of the non-text area 805. Because the non-text area 805 is a rectangular contour key point, such as its minimum and maximum positions (respectively pos_min and pos_max) can be obtained by a detection model, and the detection model can be based on the detected or identified position, as shown in output 425. For example, pos_min can be based on the position column_min and the position row_min. Pos_max can be based on the position column_max and row_max. However, the identification pos_min and pos_max may not guarantee that the recognized or detected non-text elements in the non-text area 805 are closed and all boundaries exist. In this example, there are three of the four boundaries of the non-text elements in the non-text area 805. Thus, in order to detect whether there is an open boundary, the non-text element can be reconstructed based on the output information associated with the non-text area 805, and the non-text element is a non-text element 415 in this example. When the reconstructed image is the same as the base image, the base image may not be truncated. When the reconstructed image is different from the base image, the base image may be truncated.

[0061] Fig. 9 Non-text area 905 is shown, which may be a modified image based on non-text area 805 after pre-processing. Pre-processing includes adjusting the color of the non-text elements to a monochrome tone, such as gray, before saving the image. Since the pixel colors of the non-text elements are the same, this allows for easier comparison of images. After pre-processing, non-text area 905 may be copied and based on Figure 8The contour key points identified in the image are used to draw the contour to reconstruct the truncated shape and close the open space. The reconstructed image of the non-text area 905 can then be saved as the non-text area 910. Computer vision techniques can then be used to compare the non-text area 905 used as the base image with the non-text area 910 to determine whether they are the same.

[0062] although Figure 3 In some embodiments, example blocks of method 300 are shown, but are not necessarily identical to those of Figure 3 Method 300 may include additional boxes, fewer boxes, different boxes, or boxes arranged differently than those boxes depicted in . Those skilled in the art will appreciate that the principles set forth herein may be implemented in any appropriately arranged processing system. Additionally or alternatively, two or more boxes of method 300 may be executed in parallel. For example, box 325 and box 330 of method 300 may be executed in parallel.

[0063] According to various embodiments of the present disclosure, the methods described herein can be implemented by a software program executable by a computer system. In addition, in an exemplary non-limiting embodiment, implementations can include distributed processing, component / object distributed processing, and parallel processing. Alternatively, a virtual computer system process can be constructed to implement one or more of the methods or functionalities as described herein.

[0064] The embodiments described herein may be configured as hardware when referred to as a "device," "module," "unit," "controller," etc. For example, a portion of an information handling system device may be hardware, such as an integrated circuit (such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a structured ASIC, or a device embedded on a larger chip), a card (such as a peripheral component interface (PCI) card, a PCI-express card, a personal computer memory card international association (PCMCIA) card, or other such expansion card), or a system (such as a motherboard, a system on a chip (SoC), or a stand-alone device).

[0065] The present disclosure contemplates a computer-readable medium that includes instructions or receives and executes instructions in response to a propagation signal; so that a device connected to a network can transmit voice, video or data through the network. In addition, instructions can be transmitted or received through the network via a network interface device.

[0066] Although the computer-readable medium is illustrated as a single medium, the term "computer-readable medium" includes a single medium or multiple media, such as a centralized or distributed database, and / or associated caches and servers storing one or more instruction sets. The term "computer-readable medium" shall also include any medium that can store, encode, or carry an instruction set that is executed by a processor or causes a computer system to perform any one or more of the methods or operations disclosed herein.

[0067] In a specific non-limiting exemplary embodiment, the computer-readable medium may include a solid-state memory, such as a memory card or other package that accommodates one or more non-volatile read-only memories. In addition, the computer-readable medium may be a random access memory or other volatile rewritable memory. In addition, the computer-readable medium may include a magneto-optical or optical medium (such as a disk or tape) or another storage device to store information received via a carrier signal (such as a signal transmitted by a transmission medium). The digital file attachment of an email or other independent information archive or archive set may be considered as a distributed medium equivalent to a tangible storage medium. Therefore, the present disclosure is considered to include any one or more of a computer-readable medium or distributed medium and other equivalents and successor media in which data or instructions may be stored.

[0068] Although only a few exemplary embodiments have been described in detail above, those skilled in the art will readily appreciate that many modifications may be made to the exemplary embodiments without materially departing from the novel teachings and advantages of the embodiments of the present disclosure. Therefore, all such modifications are intended to be included within the scope of the embodiments of the present disclosure as defined by the appended claims. In the claims, means-plus-function clauses are intended to cover structures described herein as performing the recited function, not only structural equivalents, but also equivalent structures.

Claims

1. A method comprising: detecting, by a processor, text elements and non-text elements in the graphical user interface; detecting whether the text element is truncated by comparing the text element in the graphical user interface to an expected text element using a pattern matching algorithm; as well as Whether the non-text element is truncated is detected by comparing the non-text element in the graphical user interface with a reconstructed copy of the non-text element. 2 . The method of claim 1 , wherein comparing the text element to the expected text element comprises filtering symbols. 3 . The method of claim 2 , wherein the filtering of the symbols is performed before performing another comparison of the text element with the expected text element when the text element is not identical to the expected text element. The method of claim 1 , wherein the expected text element is extracted from a screenshot. 5 . The method of claim 1 , wherein the reconstructed copy of the non-text element is generated by overlaying contour keypoints of the copy of the non-text element.

6. The method of claim 1, wherein the detection of the text elements is performed using a deep learning text detector.

7. The method of claim 1, wherein the detection of the non-text elements is performed using computer vision techniques.

8. An information processing system, comprising: processor; as well as a memory storing instructions that, when executed, cause the processor to perform operations including: Detect text and non-text elements in graphical user interfaces; detecting whether the text element is truncated by comparing the text element in the graphical user interface to an expected text element using a pattern matching algorithm; as well as Whether the non-text element is truncated is detected by comparing the non-text element in the graphical user interface with a reconstructed copy of the non-text element.

9. The information handling system of claim 8, wherein comparing the text element to the expected text element comprises filtering symbols.

10. The information handling system of claim 9, wherein the filtering of the symbols is performed before performing another comparison of the text element with the expected text element when the text element is not identical to the expected text element.

11. The information handling system of claim 8, wherein the expected text element is extracted from a screenshot.

12. The information handling system of claim 8, wherein the reconstructed copy of the non-text element is generated by overlaying contour keypoints of the copy of the non-text element.

13. The information handling system of claim 8, wherein comparing the text element to the expected text element comprises using a regular expression.

14. The information handling system of claim 8, wherein the detection of the text elements is performed using a deep learning text detector.

15. The information handling system of claim 8, wherein the detection of the non-text elements is performed using computer vision techniques.

16. A non-transitory computer-readable medium for storing instructions executable to perform operations comprising: detecting whether a text element in the graphical user interface is truncated by comparing the text element to an expected text element using a pattern matching algorithm; and Whether the non-text element is truncated is detected by comparing the non-text element in the graphical user interface with a reconstructed copy of the non-text element.

17. The non-transitory computer-readable medium of claim 16, wherein comparing the text element to the expected text element comprises filtering symbols.

18. The non-transitory computer-readable medium of claim 17, wherein the filtering of the symbols is performed before performing another comparison of the text element to the expected text element when the text element is not identical to the expected text element.

19. The non-transitory computer-readable medium of claim 16, wherein the expected text element is extracted from a screenshot.

20. The non-transitory computer-readable medium of claim 16, wherein the reconstructed copy of the non-text element is generated by overlaying contour keypoints of the copy of the non-text element.

Images