Graphical interface testing method and device

Abstract

The invention discloses a graphical interface testing method and device, and relates to the technical field of computer image processing. Comprising the following steps: constructing an intelligent area screenshot system by adopting a three-layer event-driven architecture, and performing screenshot and screenshot information processing by utilizing the intelligent area screenshot system; wherein the three-layer event-driven architecture comprises an event monitoring layer, a coordinate processing layer and a service integration layer; the event monitoring layer is used for capturing a screenshot signal and an event triggered by user operation and recording associated original boundary coordinate points; the coordinate processing layer is used for loading correction parameters from a configuration file, correcting original boundary coordinate points and determining boundary coordinates of a screenshot area; the service integration layer is used for executing screenshot storage and size acquisition operation according to the determined screenshot area coordinates; and integrating the operation of the intelligent area screenshot system into an automatic test system, and carrying out a graphical interface test. The problems of how to realize efficient screenshot operation and how to trace back the screenshot content are solved.

Description

Technical Field

[0001] The present application relates to the technical field of computer image processing, and in particular to a graphical interface testing method and device. Background Art

[0002] Due to differences in design philosophies and technical approaches among operating system vendors, their graphical user interfaces (GUIs) exhibit significant heterogeneity in terms of element layout, control styles, and interaction logic. Different versions from the same vendor, or different branches of the same version, also exhibit significant differences in UI (user interface) layout, control styles, and interaction logic.

[0003] Existing screenshot tools require manual startup and area selection. In automated adaptation scenarios, a large amount of manual operation is required to save the expected screenshots, and there is a lack of the ability to record and trace the screenshot content.

[0004] Therefore, how to achieve efficient screenshot operations for non-automated adapters and automated adapters in automated testing, and to trace back the content of the screenshots, becomes a problem to be solved. Summary of the Invention

[0005] In an embodiment of the present application, by providing a graphical interface testing method, the problem of how to achieve efficient screenshot operations for non-automated adapter personnel and automated adapter personnel in automated testing and to trace back the content of the screenshots is solved.

[0006] In the first aspect, an embodiment of the present application provides a graphical interface testing method, which includes: using a three-layer event-driven architecture to build an intelligent area screenshot system, and using the intelligent area screenshot system to take screenshots and process screenshot information; wherein the three-layer event-driven architecture includes an event monitoring layer, a coordinate processing layer, and a service integration layer; the event monitoring layer is used to capture screenshot signals and events triggered by user operations, and record the associated original boundary coordinate points; the coordinate processing layer is used to load correction parameters from a configuration file, correct the original boundary coordinate points, and determine the boundary coordinates of the screenshot area; the service integration layer is used to perform screenshot saving and size acquisition operations according to the determined screenshot area coordinates; the operations of the intelligent area screenshot system are integrated into the automated testing system to perform graphical interface testing.

[0007] In one possible implementation, a three-layer event-driven architecture is used to construct an intelligent area screenshot system, and the intelligent area screenshot system is used to perform screenshots and screenshot information processing, including: the operation of the event monitoring layer includes: building a global event bus based on pynput to implement a cross-process Ctrl key state synchronization mechanism; setting keyboard and mouse event monitoring modules to monitor Ctrl key trigger signals and mouse click events; when it is detected that the Ctrl key is pressed and a mouse click event occurs, recording the original boundary coordinate point of the current mouse click and marking it as the coordinate to be corrected; the operation of the coordinate processing layer includes: loading correction parameters from a configuration file, including display scaling, aspect ratio compensation, calibration offset, and DPI compensation value; correcting the coordinates to be corrected to determine the boundary coordinates of the screenshot area; wherein the boundary coordinates are the coordinates of the two diagonal vertices of the screenshot area, including the starting coordinates and the ending coordinates of the screenshot area; the expression for correcting the obtained original boundary coordinate points is: ;in, After correction coordinate, For the original coordinate, To display the zoom ratio, is the calibration offset; ;in, After correction coordinate, is the screen height, For the original coordinate, For aspect ratio compensation, is the DPI compensation value; the method for determining the boundary coordinates of the screenshot area includes: when clicking for the first time, correcting the original boundary coordinate points to obtain the corrected coordinate points, and using them as the starting coordinates of the screenshot area; when clicking for the second time, correcting the original boundary coordinate points to obtain the corrected coordinate points, and using them as the ending coordinates of the screenshot area; the operations of the service integration layer include: executing screen area capture based on the determined boundary coordinates of the screenshot area and saving it as a screenshot file; obtaining the size of the screenshot area based on the determined boundary coordinates of the screenshot area; the size of the screenshot area includes the height and width of the screenshot area; obtaining the save path of the screenshot file from the configuration, and using the default path if not configured, and ensuring that the path exists; in During the screenshot saving process, the metadata automatic generation system is started; the system includes a screenshot file MD5 verification module, an operation log recording module and a coordinate parameter persistent storage unit; the screenshot file MD5 verification module is used to read the screenshot file content, obtain the MD5 verification code of the screenshot file, and record the verification code in the metadata file corresponding to the screenshot file; the operation log recording module is used to record the key information of the operation in real time during the execution of the screenshot operation; the key information includes the start and end time of the operation and the operation type; the coordinate parameter persistent storage unit is used to associate the boundary coordinates of the screenshot area with the screenshot file and store them in the metadata file; all exceptions in the screenshot process are captured, and the error information is printed through the standard error output.

[0008] In one possible implementation, the method further includes using a hierarchical configuration management system to separate system-level and user-level configurations, specifically including: loading a configuration file according to a configuration loading priority, loading a user-level configuration file after loading a system-level configuration file, merging the system-level configuration file and the user-level configuration file according to a merging rule to obtain a merged configuration file; the merging rule includes: the user-level configuration overwrites the same configuration items in the system-level configuration, and missing items fall back to the default value; verifying the configuration items, and when the value of the configuration item is not a numeric value or exceeds a preset range, falling back to the default value; and generating a save path for the configuration file through an automatic path generation algorithm.

[0009] In one possible implementation, the operation of the smart area screenshot system is integrated into an automated testing system to perform a graphical interface test, including: the automated testing system parses a pre-configured configuration file and traverses each control option in the control configuration section of the configuration file; wherein the configuration file is used to define the control information in the graphical interface test; for each control option, obtain and parse control information that conforms to a preset format, the control information including a control identifier, a resolution of a control resource map, a path, upper left corner coordinates, lower right corner coordinates, and an MD5 checksum; the parsed resolution is verified, and if it is not in a valid resolution list, a resolution adjustment function is called to adjust the resolution of the control resource map; the parsed coordinates are verified, and if the number of coordinates is not 4 or the upper left corner coordinate is greater than or equal to the lower right corner coordinate, a coordinate error exception is thrown; the parsed and / or verified control information is stored in a control dictionary, where the dictionary key is the control identifier and the value is a dictionary including the control information; each control information in the control dictionary is traversed, and the control information is judged. Determine whether the path of each control resource map exists; if the path of the control resource map does not exist, the control positioning operation of the control resource map fails, and the resolution adjustment function is called to adjust the resolution of the control resource map; according to the upper left corner coordinates and lower right corner coordinates of the control resource map, the screenshot function is called to take a screenshot of the specified area, and the screenshot is saved to the configuration path, a new MD5 verification code is generated and the configuration file is updated; if the path of the control resource map exists, the image recognition step is executed; if the recognition is successful, the graphical interface test operation is performed according to the test case; if the recognition fails, the resolution adjustment function is called to adjust the resolution of the control resource map; according to the upper left corner coordinates and lower right corner coordinates of the control resource map, the screenshot function is called to take a new screenshot, a new screenshot is obtained, the MD5 verification code of the new screenshot is obtained, and it is compared with the MD5 verification code of the control resource map in the original configuration file; if the MD5 verification codes are consistent, the user is prompted that there is a problem with the current image recognition; if the MD5 verification codes are inconsistent, the MD5 verification code of the control resource map in the configuration file is updated.

[0010] In one possible implementation, the image recognition step includes: reading the control resource map, converting the control resource map and the current interface screenshot area into grayscale images, and performing Gaussian blur preprocessing; using multi-scale template matching technology to dynamically adjust the size of the screenshot area at multiple zoom scales, performing template matching to obtain matching results; recording the highest similarity value in the matching results of each scale, comparing it with a preset matching threshold, and determining whether the recognition is successful; if it is greater than or equal to the preset matching threshold, the recognition is successful; if it is less than the preset matching threshold, the recognition fails.

[0011] In the second aspect, an embodiment of the present application provides a graphical interface testing device, which includes: a construction module for constructing an intelligent area screenshot system using a three-layer event-driven architecture, and using the intelligent area screenshot system to take screenshots and process screenshot information; wherein, the three-layer event-driven architecture includes an event monitoring layer, a coordinate processing layer, and a service integration layer; the event monitoring layer is used to capture screenshot signals and events triggered by user operations, and record the associated original boundary coordinate points; the coordinate processing layer is used to load correction parameters from a configuration file, correct the original boundary coordinate points, and determine the boundary coordinates of the screenshot area; the service integration layer is used to perform screenshot saving and size acquisition operations according to the determined screenshot area coordinates; an integration module is used to integrate the operations of the intelligent area screenshot system into the automated testing system to perform graphical interface testing.

[0012] In a third aspect, an embodiment of the present application provides a graphical interface testing server, comprising a memory and a processor; the memory is used to store computer-executable instructions; the processor is used to execute the computer-executable instructions to implement the method described in the first aspect or any possible implementation method of the first aspect.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores executable instructions. When a computer executes the executable instructions, it can implement the method described in the first aspect or any possible implementation method of the first aspect.

[0014] One or more technical solutions provided in the embodiments of this application have at least the following technical effects: The embodiment of the present application provides a graphical interface testing method, which constructs an intelligent area screenshot system by adopting a three-layer event-driven architecture, and uses the intelligent area screenshot system to perform screenshots and screenshot information processing. The event monitoring layer is used to capture the screenshot signals and events triggered by user operations, and record the associated original boundary coordinate points, solving the positioning failure problem of traditional screenshot tools due to dynamic loading of controls. The coordinate processing layer is used to load correction parameters from the configuration file, correct the original boundary coordinate points, and determine the boundary coordinates of the screenshot area, which can effectively improve the accuracy of the screenshot. The service integration layer is used to perform screenshot saving and size acquisition operations based on the determined screenshot area coordinates. The operation of the intelligent area screenshot system is integrated into the automated testing system for graphical interface testing. How to achieve efficient screenshot operations for non-automated adapters and automated adapters in automated testing, and to trace back the content of the screenshot, has become a problem to be solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments of the present application or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 A flowchart of a graphical interface testing method provided in an embodiment of the present application; Figure 2 A schematic diagram of a graphical interface testing device provided in an embodiment of the present application; Figure 3 A schematic diagram of a graphical interface test server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0018] The following description of some of the technologies involved in the embodiments of this application is provided to facilitate understanding and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for the sake of clarity and conciseness, some descriptions of well-known functions and structures are omitted from the following description.

[0019] The present application embodiment provides a graphical interface testing method, such as Figure 1 As shown, the method includes steps S101 to S102. Figure 1 This is only an execution order shown in the embodiment of the present application, and does not represent the only execution order of a graphical interface testing method. If the final result can be achieved, Figure 1 The steps shown may be performed in parallel or reversed.

[0020] S101: A three-layer event-driven architecture is used to build an intelligent area screenshot system, which is used to take screenshots and process screenshot information. The three-layer event-driven architecture includes an event monitoring layer, a coordinate processing layer, and a service integration layer.

[0021] The event monitoring layer captures screenshot signals and events triggered by user actions and records the associated original boundary coordinates. The coordinate processing layer loads correction parameters from the configuration file, corrects the original boundary coordinates, and determines the boundary coordinates of the screenshot area. The service integration layer saves the screenshot and retrieves its size based on the determined screenshot area coordinates.

[0022] A three-tier event-driven architecture is used to build an intelligent area screenshot system, which is used to take screenshots and process screenshot information, including the following contents.

[0023] The event monitoring layer operates by building a global event bus based on Pynput to synchronize the Ctrl key state across processes. Keyboard and mouse event monitoring modules are set up to listen for Ctrl key trigger signals and mouse click events. When a Ctrl key press and mouse click event are detected, the original boundary coordinates of the mouse click are recorded and marked as coordinates to be corrected.

[0024] Specifically, pynpu is a library in Python specifically used to control and monitor input devices.

[0025] The coordinate processing layer loads calibration parameters from a configuration file, including display scaling, aspect ratio compensation, calibration offset, and DPI compensation. It then calibrates the coordinates to determine the boundary coordinates of the screenshot area. These boundary coordinates are the coordinates of the two diagonal vertices of the screenshot area, including the start and end coordinates.

[0026] The expression for correcting the original boundary coordinate points is: .in, After correction coordinate, For the original coordinate, To display the zoom ratio, is the calibration offset. .in, After correction coordinate, is the screen height, For the original coordinate, For aspect ratio compensation, It is the DPI compensation value.

[0027] Specifically, the coordinates to be corrected can be corrected. In this application, the coordinates to be corrected are clicked twice to define the starting coordinates and the ending coordinates of the screenshot area.

[0028] The boundary coordinates of the screenshot area are determined by: upon the first click, the original boundary coordinates are corrected to obtain the corrected coordinates, which are used as the starting coordinates of the screenshot area. Upon the second click, the original boundary coordinates are also corrected to obtain the corrected coordinates, which are used as the ending coordinates of the screenshot area.

[0029] Specifically, the first click determines the starting point (start_point), and the second click determines the ending point (end_point). Because the user can drag in any direction (e.g., from the lower right to the upper left), directly using (start_x, start_y) and (end_x, end_y) as the diagonal vertices of the rectangle will result in incorrect area calculation (e.g., the area of ​​the rectangle is negative when the coordinates are reversed). Where start_x is the abscissa of the starting point determined by the first click, start_y is the ordinate of the starting point determined by the first click, end_x is the abscissa of the ending point determined by the second click, and end_y is the ordinate of the ending point determined by the second click.

[0030] Therefore, we need to use the min and max functions to normalize the coordinates to ensure that the screenshot area always contains a valid rectangle. The min function is used to find the minimum value, and the max function is used to find the maximum value. These two functions automatically obtain a valid rectangular area regardless of how the user drags the mouse. That is, the actual screenshot area is a rectangle from (min(start_x, end_x), min(start_y, end_y)) to (max(start_x, end_x), max(start_y, end_y)).

[0031] The operation of the service integration layer includes: executing screen area capture based on the determined boundary coordinates of the screenshot area and saving the captured image as a screenshot file.

[0032] The size of the screenshot area is obtained based on the determined boundary coordinates of the screenshot area. The size of the screenshot area includes the height and width of the screenshot area.

[0033] Get the path for saving screenshot files from the configuration. If not configured, use the default path and ensure that the path exists.

[0034] Specifically, the present application can also use the timestamp to generate a unique screenshot file name. The format of the screenshot file can be a PNG file.

[0035] During the screenshot saving process, the metadata automatic generation system is started. The system includes a screenshot file MD5 verification module, an operation log recording module, and a coordinate parameter persistent storage unit.

[0036] The screenshot file MD5 verification module is used to read the content of the screenshot file, obtain the MD5 verification code of the screenshot file, and record this verification code in the metadata file corresponding to the screenshot file.

[0037] Specifically, the suffix of the metadata file is.meta and it has the same name as the screenshot file. The screenshot file MD5 verification module is specifically used to read the binary content of the screenshot file and obtain the MD5 verification code of the screenshot file.

[0038] The operation log recording module is used to record the key information of the operation in real time during the execution of the screenshot operation. The key information includes the start and end times of the operation and the operation type.

[0039] Specifically, when the operation time is not explicitly recorded, the start and end times of the operation can be added to the metadata file. The operation type is defaulted to area screenshot.

[0040] The coordinate parameter persistent storage unit is used to associate and store the boundary coordinates of the screenshot area with the screenshot file in the metadata file.

[0041] Capture all exceptions during the screenshot process and print the error information through the standard error output.

[0042] Specifically, this application captures all exceptions through try-except (direct annotation method), such as non-existent path, permission error, screenshot failure, etc.

[0043] Furthermore, for the two corrected coordinates, that is, the starting coordinates (x1, y1) of the screenshot area and the ending coordinates (x2, y2) of the screenshot area. Use the sorted function (a function for sorting) to sort the abscissa x1 and ordinate y1 of the starting coordinates of the screenshot area and the abscissa x2 and ordinate y2 of the ending coordinates of the screenshot area to ensure x1 < x2 and y1 < y2, so as to obtain a standard rectangular screenshot area. The width width of the screenshot area is width = x2 - x1, and the height height of the screenshot area is height = y2 - y1.

[0044] Furthermore, the coordinates of the valid rectangular area are represented by (x1, y1, width, height). When x1 = 100, y1 = 50, width = 200, and height = 150, it means that the upper left corner of the screenshot area is at the screen coordinates (100, 50), the rectangle extends 200 pixels to the right and 150 pixels downwards, and the lower right corner coordinates are (x1 + width, y1 + height) = (300, 200).

[0045] This application also includes using a hierarchical configuration management system to separate system-level and user-level configurations, specifically including the following content.

[0046] The configuration file is loaded according to the configuration loading priority. After the system-level configuration file is loaded, the user-level configuration file is loaded. The system-level configuration file and the user-level configuration file are merged according to a merging rule to obtain a merged configuration file.

[0047] The merging rules include: user-level configurations overwrite identical configuration items in the system-level configurations, and missing items fall back to default values.

[0048] Specifically, this application uses the configparser function (a function for merging) to implement merging.

[0049] Verify the configuration item. If the value of the configuration item is not a numeric value or exceeds the preset range, it will fall back to the default value.

[0050] It should be noted that the default value is the resolution of the current test environment. For example, you can use two independent command-line tools, xrandr and gsettings, on Linux to obtain the resolution of the current test environment.

[0051] Generate the save path of the configuration file through the automatic path generation algorithm.

[0052] The automatic path generation algorithm includes: generating user-specific subdirectories based on the system user ID (such as os.getuid() in Linux), organizing screenshot files by year / month / day hierarchy, and creating directories to ensure the path exists.

[0053] Specifically, in this application, the user only needs to press the Ctrl key to trigger the screenshot, and only two clicks (upper left corner and lower right corner) are required to complete the screenshot. The system automatically processes the relevant information of the saved screenshot file.

[0054] S102: Integrate the operation of the intelligent area screenshot system into the automated testing system to perform graphical interface testing.

[0055] Integrate the operation of the intelligent area screenshot system into the automated testing system and perform graphical interface testing, including the following contents.

[0056] The automated test system parses the pre-configured configuration file and traverses the various control options in the control configuration section of the configuration file. Among them, the configuration file is used to define the control information in the graphical interface test.

[0057] For each control option, obtain and parse control information that conforms to a preset format. The control information includes a control identifier, a resolution of a control resource map, a path, upper left corner coordinates, lower right corner coordinates, and an MD5 checksum.

[0058] Specifically, the control resource diagram is a standardized visual template for controls, usually an image file in PNG / BMP format.

[0059] Verify the parsed resolution. If it is not in the valid resolution list, call the resolution adjustment function to adjust the resolution of the control resource diagram.

[0060] Specifically, the valid resolution list is obtained by parsing the configuration file, which is the resolution of each control resource diagram in the configuration file. Adjust the resolution to be the same as the resolution of the current test environment.

[0061] Specifically, the resolution adjustment function can be resolution_adjust().

[0062] Verify the parsed coordinates. If the number of coordinates is not 4 or the upper left coordinate is greater than or equal to the lower right coordinate, throw a coordinate error exception.

[0063] Specifically, the parsed coordinates can be (x1, y1, x2, y2), which need to satisfy x1 < x2 and y1 < y2, otherwise throw CoordinatesError, that is, a coordinate error exception.

[0064] Store the parsed and / or verified control information in the control dictionary, where the dictionary key is the control identifier and the value is a dictionary including the control information.

[0065] Specifically, the control dictionary of this application is controls[key].

[0066] Traverse each control information in the control dictionary and determine whether the path of each control resource diagram exists.

[0067] If the path of the control resource diagram does not exist, the control positioning operation of the control resource diagram fails. Call the resolution adjustment function to adjust the resolution of the control resource diagram. According to the upper left coordinate and the lower right coordinate of the control resource diagram, call the screenshot function to take a screenshot of the specified area and save the screenshot to the configuration path, generate a new MD5 checksum and update the configuration file.

[0068] If the path of the control resource diagram exists, perform the image recognition step.

[0069] Specifically, the image recognition step includes the following content.

[0070] Read the control resource diagram, convert both the control resource diagram and the current interface screenshot area to grayscale images, and perform Gaussian blur preprocessing.

[0071] Specifically, the Gaussian blur preprocessing in this application can be kernel = 3×3. That is, the size of the convolution kernel kernel is 3×3.

[0072] Multi-scale template matching technology is used to dynamically adjust the size of the screenshot area at multiple zoom scales, and template matching is performed to obtain matching results.

[0073] Specifically, the multiple scaling scales may be [0.9, 1.0, 1.1].

[0074] The highest similarity value among the matching results at each scale is recorded and compared with the preset matching threshold to determine whether the recognition is successful.

[0075] Specifically, the preset matching threshold may be 0.75.

[0076] Specifically, the similarity value of the matching result in this application can be obtained through the cv2.matchTemplate() function (template matching function) combined with the cv2.TM_CCOEFF_NORMED method (normalized correlation coefficient matching method).

[0077] If it is greater than or equal to the preset matching threshold, the recognition is successful.

[0078] If it is less than the preset matching threshold, the recognition fails.

[0079] If the recognition is successful, perform graphical interface test operations according to the test case.

[0080] If recognition fails, call the resolution adjustment function to adjust the resolution of the control resource map. Based on the coordinates of the upper left and lower right corners of the control resource map, call the screenshot function to take a new screenshot, obtain the MD5 checksum of the new screenshot, and compare it with the MD5 checksum of the control resource map in the original configuration file.

[0081] If the MD5 checksums are consistent, the user will be prompted that there is a problem with the current image recognition.

[0082] If the MD5 checksums are inconsistent, update the MD5 checksums of the control resource graph in the configuration file.

[0083] This application integrates the operation of the smart area screenshot system into the automated testing system to perform graphical interface testing, solving the problem of automated testing requiring re-adaptation due to slight changes in the screenshots.

[0084] The present application also provides a graphical interface testing device 200, such as Figure 2 As shown, the device includes: a building module 201 and an integration module 202.

[0085] Construction module 201 is used to construct an intelligent area screenshot system using a three-layer event-driven architecture, and utilize the intelligent area screenshot system to take screenshots and process screenshot information. The three-layer event-driven architecture includes an event monitoring layer, a coordinate processing layer, and a service integration layer. The event monitoring layer is used to capture screenshot signals and events triggered by user operations and record the associated original boundary coordinate points. The coordinate processing layer is used to load correction parameters from a configuration file, correct the original boundary coordinate points, and determine the boundary coordinates of the screenshot area. The service integration layer is used to perform screenshot saving and size acquisition operations based on the determined screenshot area coordinates.

[0086] The integration module 202 is used to integrate the operation of the intelligent area screenshot system into the automated testing system to perform graphical interface testing.

[0087] Some modules in the apparatus described herein may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0088] The devices or modules described in the above application embodiments can be implemented by computer chips or physical devices, or by products with certain functions. For ease of description, the above devices are described separately by function in various modules. When implementing the embodiments of this application, the functions of each module can be implemented in the same or multiple software and / or hardware. Of course, a module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0089] The methods, devices, or modules described herein can be implemented in the form of computer-readable program code. The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also appreciate that, in addition to implementing the controller in the form of pure computer-readable program code, the controller can also be implemented in the form of logic gates, switches, an application-specific integrated circuit, a programmable logic controller, an embedded microcontroller, etc. by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the means for implementing various functions may be considered to be both a software module for implementing the method and a structure within a hardware component.

[0090] like Figure 3 As shown, an embodiment of the present application also provides a graphical interface testing server, including a memory 301 and a processor 302; the memory 301 is used to store computer-executable instructions; the processor 302 is used to execute computer-executable instructions to implement a graphical interface testing method described above in an embodiment of the present application.

[0091] An embodiment of the present application further provides a computer-readable storage medium, which stores executable instructions. When a computer executes the executable instructions, it can implement a graphical interface testing method described above in the embodiment of the present application.

[0092] Through the description of the above implementation methods, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, or can be embodied through the implementation process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the method described in the embodiments of the present application.

[0093] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to in detail. Each embodiment focuses on the differences from other embodiments. All or part of this application can be used in many general or special computer system environments or configurations.

[0094] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some or all of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present application.

Claims

1. A graphical interface testing method, characterized in that: include: A three-tier event-driven architecture is used to build an intelligent area screenshot system, which is used to take screenshots and process screenshot information. The three-tier event-driven architecture includes an event monitoring layer, a coordinate processing layer, and a service integration layer. The event monitoring layer is used to capture screenshot signals and events triggered by user operations and record the associated original boundary coordinate points; The coordinate processing layer is used to load the correction parameters from the configuration file, correct the original boundary coordinate points, and determine the boundary coordinates of the screenshot area; The service integration layer is used to perform screenshot saving and size acquisition operations based on the determined screenshot area coordinates; Integrate the operation of the intelligent area screenshot system into the automated testing system to perform graphical interface testing.

2. The graphical interface testing method according to claim 1, wherein: The three-layer event-driven architecture is used to build an intelligent area screenshot system, and the intelligent area screenshot system is used to perform screenshots and screenshot information processing, including: The operations of the event monitoring layer include: building a global event bus based on pynput to implement a cross-process Ctrl key state synchronization mechanism; setting up keyboard and mouse event monitoring modules to listen for Ctrl key trigger signals and mouse click events; when the Ctrl key is pressed and a mouse click event occurs, recording the original boundary coordinates of the current mouse click and marking them as coordinates to be corrected; The coordinate processing layer operates by loading correction parameters from a configuration file, including display scaling, aspect ratio compensation, calibration offset, and DPI compensation; correcting the coordinates to be corrected to determine the boundary coordinates of the screenshot area; wherein the boundary coordinates are the coordinates of the two diagonal vertices of the screenshot area, including the starting coordinates and the ending coordinates of the screenshot area; The expression for correcting the original boundary coordinate points is: ;in, After correction coordinate, For the original coordinate, To display the zoom ratio, is the calibration offset; ;in, After correction coordinate, is the screen height, For the original coordinate, For aspect ratio compensation, is the DPI compensation value; The method for determining the boundary coordinates of the screenshot area includes: when clicking for the first time, correcting the original boundary coordinate points to obtain the corrected coordinate points, and using them as the starting coordinates of the screenshot area; when clicking for the second time, correcting the original boundary coordinate points to obtain the corrected coordinate points, and using them as the ending coordinates of the screenshot area; The operations of the service integration layer include: executing screen area capture based on the determined boundary coordinates of the screenshot area and saving it as a screenshot file; Acquire the size of the screenshot area based on the determined boundary coordinates of the screenshot area; the size of the screenshot area includes the height and width of the screenshot area; Get the path for saving the screenshot file from the configuration. If not configured, use the default path and ensure that the path exists. During the screenshot saving process, the metadata automatic generation system is started; the system includes a screenshot file MD5 verification module, an operation log recording module and a coordinate parameter persistent storage unit; The screenshot file MD5 verification module is used to read the screenshot file content, obtain the MD5 verification code of the screenshot file, and record the verification code in the metadata file corresponding to the screenshot file; The operation log recording module is used to record key information of the operation in real time during the execution of the screenshot operation; the key information includes the start and end time of the operation and the operation type; The coordinate parameter persistent storage unit is used to associate the boundary coordinates of the screenshot area with the screenshot file and store them in the metadata file; Catch all exceptions during the screenshot process and print error information through standard error output.

3. The graphical interface testing method according to claim 2, characterized in that: It also includes the use of a hierarchical configuration management system to separate system-level and user-level configurations, including: The configuration file is loaded according to the configuration loading priority. After the system-level configuration file is loaded, the user-level configuration file is loaded. The system-level configuration file and the user-level configuration file are merged according to the merging rule to obtain a merged configuration file. The merging rules include: user-level configurations overwrite identical configuration items in the system-level configurations, and missing items fall back to default values; Verify the configuration items. If the value of a configuration item is not a numeric value or exceeds the preset range, it will fall back to the default value. Generate the save path of the configuration file through the automatic path generation algorithm.

4. The graphical interface testing method according to claim 3, characterized in that: The operation of the intelligent area screenshot system is integrated into the automated testing system to perform graphical interface testing, including: The automated test system parses the pre-configured configuration file and traverses the various control options in the control configuration section of the configuration file; wherein the configuration file is used to define the control information in the graphical interface test; For each control option, obtain and parse the control information that conforms to the preset format. The control information includes the control identifier, the resolution of the control resource map, the path, the upper left corner coordinates, the lower right corner coordinates, and the MD5 checksum. Verify the parsed resolution. If it is not in the valid resolution list, call the resolution adjustment function to adjust the resolution of the control resource map. Verify the parsed coordinates. If the number of coordinates is not 4 or the upper left corner coordinate is greater than or equal to the lower right corner coordinate, throw a coordinate error exception. Store the parsed and / or verified control information in a control dictionary, where the dictionary key is the control identifier and the value is a dictionary containing the control information; Traverse each control information in the control dictionary to determine whether the path of each control resource map exists; If the control resource map path does not exist, the control positioning operation of the control resource map fails, and the resolution adjustment function is called to adjust the resolution of the control resource map. Based on the coordinates of the upper left corner and lower right corner of the control resource map, the screenshot function is called to take a screenshot of the specified area and save the screenshot to the configuration path. A new MD5 checksum is generated and the configuration file is updated. If the path of the control resource map exists, perform the image recognition step; If the recognition is successful, perform graphical interface test operations according to the test case; If recognition fails, call the resolution adjustment function to adjust the resolution of the control resource map; call the screenshot function to take a new screenshot based on the coordinates of the upper left corner and lower right corner of the control resource map, obtain the MD5 checksum of the new screenshot, and compare it with the MD5 checksum of the control resource map in the original configuration file; If the MD5 checksums are consistent, the user will be prompted that there is a problem with the current image recognition; If the MD5 checksums are inconsistent, update the MD5 checksums of the control resource graph in the configuration file.

5. The graphical interface testing method according to claim 4, characterized in that: The image recognition step comprises: Read the control resource map, convert the control resource map and the current interface screenshot area into grayscale images, and perform Gaussian blur preprocessing; Adopting multi-scale template matching technology, the size of the screenshot area is dynamically adjusted at multiple zoom scales, and template matching is performed to obtain matching results; Record the highest similarity value among the matching results at each scale, compare it with the preset matching threshold, and determine whether the recognition is successful; If it is greater than or equal to the preset matching threshold, the recognition is successful; If it is less than the preset matching threshold, the recognition fails.

6. A graphical interface testing device, characterized in that: include: A construction module is used to construct an intelligent area screenshot system using a three-layer event-driven architecture, and utilize the intelligent area screenshot system to take screenshots and process screenshot information. The three-layer event-driven architecture includes an event monitoring layer, a coordinate processing layer, and a service integration layer. The event monitoring layer is used to capture screenshot signals and events triggered by user operations and record the associated original boundary coordinate points. The coordinate processing layer is used to load correction parameters from a configuration file, correct the original boundary coordinate points, and determine the boundary coordinates of the screenshot area. The service integration layer is used to perform screenshot saving and size acquisition operations based on the determined screenshot area coordinates. The integration module is used to integrate the operation of the intelligent area screenshot system into the automated testing system for graphical interface testing.

7. A graphical interface test server, characterized in that: including memory and processor; The memory is used to store computer-executable instructions; The processor is configured to execute the computer-executable instructions to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores executable instructions, and when a computer executes the executable instructions, the method according to any one of claims 1 to 5 can be implemented.

Images