A graphical interface testing method and apparatus

Abstract

The application discloses a graphical interface testing method and device, and relates to the technical field of computer image processing. The method comprises the following steps: constructing an intelligent region screenshot system by adopting a three-layer event-driven architecture, and performing screenshot and screenshot information processing by using the intelligent region screenshot system; wherein the three-layer event-driven architecture comprises an event listening layer, a coordinate processing layer and a service integration layer; the event listening layer is used for capturing a screenshot signal and an event triggered by a user operation, and recording associated original boundary coordinate points; the coordinate processing layer is used for loading correction parameters from a configuration file, correcting the original boundary coordinate points, and determining boundary coordinates of a screenshot region; the service integration layer is used for performing screenshot saving and size acquisition operations according to the determined screenshot region coordinates; and the operation of the intelligent region screenshot system is integrated into an automatic test system to perform graphical interface testing. The problems of how to realize efficient screenshot operation and how to realize content backtracking of the screenshot are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer image processing, and particularly relates to a graphical interface testing method and device. BACKGROUND

[0002] Due to differences in design concepts and technical routes of different operating system manufacturers, graphical interfaces (GUIs) of different operating systems present significant heterogeneity in element layout, control style and interaction logic. Different versions of the same manufacturer or different branches of the same version also present significant differences in UI (user interface) layout, control style and interaction logic.

[0003] Existing screenshot tools need to be manually started and the region needs to be selected, and in the automatic adaptation scene, a large amount of manual operation is needed to save the expected screenshot, and the recording and backtracking ability of the screenshot content is also lacking.

[0004] Therefore, how to realize efficient screenshot operation of non-automatic adaptation personnel and automatic adaptation personnel in automatic testing, and realize the backtracking of the content of the screenshot, becomes a problem to be solved. SUMMARY

[0005] In the embodiments of the present application, by providing a graphical interface testing method, the problem of how to realize efficient screenshot operation of non-automatic adaptation personnel and automatic adaptation personnel in automatic testing, and realize the backtracking of the content of the screenshot, is solved.

[0006] In a first aspect, the embodiments of the present application provide a graphical interface testing method, which comprises: constructing an intelligent region screenshot system by using a three-layer event-driven architecture, and performing screenshot and screenshot information processing by using the intelligent region screenshot system; wherein the three-layer event-driven architecture comprises an event listening layer, a coordinate processing layer and a service integration layer; the event listening layer is used for capturing a screenshot signal and an event triggered by a user operation, and recording associated original boundary coordinate points; the coordinate processing layer is used for loading correction parameters from a configuration file, correcting the original boundary coordinate points, and determining boundary coordinates of a screenshot region; the service integration layer is used for performing screenshot saving and size acquisition operations according to the determined screenshot region coordinates; and the operation of the intelligent region screenshot system is integrated into an automatic testing system to perform graphical interface testing.

[0007] In a possible implementation, the intelligent region screenshot system is constructed by using a three-layer event-driven architecture, and screenshot and screenshot information processing are performed by using the intelligent region screenshot system. The operation of the event listening layer includes: constructing a global event bus based on pynput to implement a cross-process Ctrl key state synchronization mechanism; setting a keyboard and mouse event listening module to listen to a Ctrl key trigger signal and a mouse click event; when it is detected that the Ctrl key is pressed and the mouse click event occurs, recording original boundary coordinate points of the current mouse click, and marking the original boundary coordinate points as to-be-corrected coordinates. The operation of the coordinate processing layer includes: loading correction parameters from a configuration file, including a display scaling ratio, a width-to-height ratio compensation, a calibration offset, and a DPI compensation value; correcting the to-be-corrected coordinates to determine boundary coordinates of a screenshot region; the boundary coordinates are two diagonal vertex coordinates of the screenshot region, including starting coordinates and ending coordinates of the screenshot region; and an expression for correcting the obtained original boundary coordinate points is as follows: ; wherein, is the corrected coordinate, is the original coordinate, is the display scaling ratio, is the calibration offset. ; wherein, is the corrected coordinate, is a screen height, is the original coordinate, is the width-to-height ratio compensation, The DPI compensation value is determined as follows: when the first click is detected, the original boundary coordinate point is corrected to obtain a corrected coordinate point, which is used as the starting coordinate of the screenshot region; when the second click is detected, the original boundary coordinate point is also corrected to obtain a corrected coordinate point, which is used as the ending coordinate of the screenshot region. The operation of the service integration layer includes: performing screen region capture based on the determined boundary coordinates of the screenshot region and saving the captured screen region as a screenshot file; obtaining the size of the screenshot region based on the determined boundary coordinates of the screenshot region; the size of the screenshot region includes the height and width of the screenshot region; obtaining the saving path of the screenshot file from the configuration, using a default path if the configuration is not available, and ensuring that the path exists; starting a metadata automatic generation system during the screenshot saving process; the system includes a screenshot file MD5 verification module, an operation log recording module, and a coordinate parameter persistent storage unit; the screenshot file MD5 verification module is used to read the content of the screenshot file, obtain the MD5 verification code of the screenshot file, and record the verification code in the metadata file corresponding to the screenshot file; the operation log recording module is used to record the key information of the operation in real time during the execution of the screenshot operation; the key information includes the operation start and end time and the operation type; the coordinate parameter persistent storage unit is used to store the boundary coordinates of the screenshot region in association with the screenshot file in the metadata file; all exceptions during the capture of the screenshot are printed in the form of error information through standard error output.

[0008] In a possible implementation, the system further comprises using a hierarchical configuration management system to separate system-level and user-level configurations, specifically comprising: loading configuration files according to a configuration loading priority, loading a system-level configuration file first, then loading a user-level configuration file, and merging the system-level configuration file and the user-level configuration file according to a merging rule to obtain a merged configuration file; the merging rule comprises: the user-level configuration covers the same configuration item in the system-level configuration, and a missing item is rolled back to a default value; the configuration item is verified, and when the value of the configuration item is not a numerical value or exceeds a preset range, the configuration item is rolled back to a default value; and a saving path of the configuration file is generated through an automatic path generation algorithm.

[0009] In a possible implementation, the operation of the intelligent area screenshot system is integrated into the automatic test system to perform graphical interface testing, including: the automatic test system parses a pre-configured configuration file, and traverses each control option of a control configuration section in the configuration file; the configuration file is used to define control information in graphical interface testing; for each control option, control information conforming to a preset format is obtained and parsed, the control information including a control identifier, a resolution, a path, top-left coordinates, bottom-right coordinates, and an MD5 check code of a control resource image; the parsed resolution is checked, if not in a valid resolution list, a resolution adjustment function is called to adjust the resolution of the control resource image; the parsed coordinates are checked, if the number of coordinates is not 4 or the top-left coordinates are greater than or equal to the bottom-right coordinates, a coordinate error exception is thrown; the parsed and / or checked control information is stored in a control dictionary, the dictionary key being the control identifier and the value being a dictionary including the control information; each control information in the control dictionary is traversed, and it is determined whether the path of each control resource image exists; if the path of the control resource image does not exist, control positioning operation of the control resource image fails, and the resolution adjustment function is called to adjust the resolution of the control resource image; according to the top-left coordinates and the bottom-right coordinates of the control resource image, a screenshot function is called to take a screenshot of the specified area, and the screenshot is saved to a configuration path, a new MD5 check code is generated and the configuration file is updated; if the path of the control resource image exists, an image recognition step is performed; if the recognition is successful, graphical interface testing operation is performed according to a test case execution; if the recognition fails, the resolution adjustment function is called to adjust the resolution of the control resource image; according to the top-left coordinates and the bottom-right coordinates of the control resource image, the screenshot function is called to take a new screenshot, a new MD5 check code of the new screenshot is obtained, and the new MD5 check code is compared with the MD5 check code of the control resource image in the original configuration file; if the MD5 check codes are consistent, the user is prompted that there is a problem in current image recognition; if the MD5 check codes are inconsistent, the MD5 check code of the control resource image in the configuration file is updated.

[0010] In a possible implementation, the image recognition step includes: reading the control resource image, and converting the control resource image and the current interface screenshot area into grayscale images and performing Gaussian blur preprocessing; a multi-scale template matching technology is used to dynamically adjust the size of the screenshot area at multiple scaling scales, to perform template matching to obtain a matching result; a highest similarity value in each scale matching result is recorded, and compared with a preset matching threshold to determine whether the recognition is successful; if greater than or equal to the preset matching threshold, the recognition is successful; if less than the preset matching threshold, the recognition fails.

[0011] In a second aspect, an embodiment of the present application provides a graphical interface testing device, the device comprising: a construction module configured to construct an intelligent region screenshot system using a three-layer event-driven architecture, and configured to perform screenshot and screenshot information processing using the intelligent region screenshot system; wherein the three-layer event-driven architecture comprises an event listening layer, a coordinate processing layer, and a service integration layer; the event listening layer is configured to capture screenshot signals and events triggered by user operations, and record associated original boundary coordinate points; the coordinate processing layer is configured to load correction parameters from a configuration file, correct the original boundary coordinate points, and determine boundary coordinates of a screenshot region; the service integration layer is configured to perform screenshot saving and size acquisition operations according to the determined screenshot region coordinates; and an integration module configured to integrate operations of the intelligent region screenshot system into an automated testing system, and perform graphical interface testing.

[0012] In a third aspect, an embodiment of the present application provides a graphical interface testing server, comprising a memory and a processor; the memory is configured to store computer executable instructions; and the processor is configured to execute the computer executable instructions to implement the method in the first aspect or any possible implementation manner of the first aspect.

[0013] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium stores executable instructions, and a computer executes the executable instructions to implement the method in the first aspect or any possible implementation manner of the first aspect.

[0014] The one or more technical solutions provided in the embodiments of the present application have at least the following technical effects:

[0015] The embodiments of the present application provide a graphical interface testing method, which constructs an intelligent region screenshot system using a three-layer event-driven architecture, and performs screenshot and screenshot information processing using the intelligent region screenshot system. The event listening layer is configured to capture screenshot signals and events triggered by user operations, and record associated original boundary coordinate points, thereby solving the positioning failure problem of a traditional screenshot tool caused by dynamic loading of a control. The coordinate processing layer is configured to load correction parameters from a configuration file, correct the original boundary coordinate points, and determine boundary coordinates of a screenshot region, thereby effectively improving the accuracy of the screenshot. The service integration layer is configured to perform screenshot saving and size acquisition operations according to the determined screenshot region coordinates. The operations of the intelligent region screenshot system are integrated into an automated testing system, and graphical interface testing is performed. The problem of how to realize efficient screenshot operations of non-automated adaptation personnel and automated adaptation personnel in automated testing, and realize backtracking of the content of the screenshot, is solved. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application or the prior art. Obviously, the drawings described below are some embodiments of the present application, and other drawings can be obtained by those of ordinary skill in the art without creative effort under the premise of the drawings.

[0017] Figure 1 A flow chart of a graphical interface testing method provided by an embodiment of the present application;

[0018] Figure 2 A schematic diagram of a graphical interface testing device provided by an embodiment of the present application;

[0019] Figure 3 A schematic diagram of a graphical interface testing server provided by an embodiment of the present application. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort fall within the scope of the present application.

[0021] The following describes some technologies related to the embodiments of the present application to help understanding, which should be considered only as exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Also, for clarity and conciseness, the following description omits the description of some well-known functions and structures.

[0022] The embodiments of the present application provide a graphical interface testing method, as shown in the figure, the method comprises steps S101 to S102. Among them, Figure 1 Only one execution order is shown for the embodiments of the present application, which does not represent the only execution order of a graphical interface testing method, and Figure 1 the steps shown can be executed in parallel or reversed. Figure 1

[0023] S101: An intelligent region screenshot system is constructed by using a three-layer event-driven architecture, and screenshot and screenshot information processing are performed by using the intelligent region screenshot system. Among them, the three-layer event-driven architecture includes an event listening layer, a coordinate processing layer and a service integration layer.

[0024] ​The event listening layer is used to capture the screenshot signal and events triggered by user operations, and record the associated original boundary coordinate points. The coordinate processing layer is used to load the correction parameters from the configuration file, correct the original boundary coordinate points, and determine the boundary coordinates of the screenshot area. The service integration layer is used to perform screenshot saving and size acquisition operations according to the determined screenshot area coordinates.

[0025] An intelligent region screenshot system is constructed using a three-layer event-driven architecture. The intelligent region screenshot system is used for screenshot and screenshot information processing, including the following.

[0026] The operation of the event listening layer includes: building a global event bus based on pynput to realize a cross-process Ctrl key state synchronization mechanism. Set up keyboard and mouse event listening modules to listen to Ctrl key trigger signals and mouse click events. When a Ctrl key press and a mouse click event are detected, record the original boundary coordinate points of the current mouse click and mark them as to-be-corrected coordinates.

[0027] Specifically, pynput is a library in Python that is specifically used to control and listen to input devices.

[0028] The operation of the coordinate processing layer includes: loading correction parameters from the configuration file, including display scaling, aspect ratio compensation, calibration offset, and DPI compensation value. Correct the to-be-corrected coordinates to determine the boundary coordinates of the screenshot area. The boundary coordinates are the coordinates of the two opposite corners of the screenshot area, including the starting coordinates and the ending coordinates of the screenshot area.

[0029] The expression for correcting the obtained original boundary coordinate points is: wherein, is the corrected coordinate, is the original coordinate, is the display scaling, is the calibration offset. wherein, is the corrected coordinate, is the screen height, is the original coordinate, is the aspect ratio compensation, is the DPI compensation value.

[0030] Specifically, correcting the to-be-corrected coordinates can correct the to-be-corrected coordinates. The starting coordinates and the ending coordinates of the screenshot area are defined by the two clicked to-be-corrected coordinates.

[0031] The determination of the boundary coordinates of the screenshot area includes: when the first click, correcting the obtained original boundary coordinate points to obtain corrected coordinate points, which are taken as the starting coordinates of the screenshot area; when the second click, also correcting the original boundary coordinate points to obtain corrected coordinate points, which are taken as the ending coordinates of the screenshot area.

[0032] Specifically, the first click determines a starting point (start_point), and the second click determines an ending point (end_point). Since the user can drag from any direction (such as from the lower right to the upper left), directly using (start_x, start_y) and (end_x, end_y) as the diagonal vertices of the rectangle can cause errors in area calculation (such as negative area of the rectangle when the coordinates are reversed). Wherein, start_x is the horizontal coordinate of the starting point determined by the first click, start_y is the vertical coordinate of the starting point determined by the first click, end_x is the horizontal coordinate of the ending point determined by the second click, and end_y is the vertical coordinate of the ending point determined by the second click.

[0033] Therefore, the coordinates need to be normalized by the min function and the max function to ensure that the boundary of the screenshot area is always a valid rectangle. The min function is a minimum value function, and the max function is a maximum value function. Through these two functions, no matter how the user drags the mouse, a valid rectangular area can be automatically obtained. That is, the actual screenshot area is a rectangle from (min(start_x, end_x), min(start_y, end_y)) to (max(start_x, end_x), max(start_y, end_y)).

[0034] The operation of the service integration layer includes performing screen area capture based on the determined boundary coordinates of the screenshot area and saving as a screenshot file.

[0035] The size of the screenshot area is obtained based on the determined boundary coordinates of the screenshot area. The size of the screenshot area includes the height and width of the screenshot area.

[0036] The saving path of the screenshot file is obtained from the configuration, and if not configured, the default path is used, and it is ensured that the path exists.

[0037] Specifically, the application can also use a timestamp to generate a unique screenshot file name. The format of the screenshot file can be a PNG file.

[0038] During the screenshot saving process, a metadata automatic generation system is started. The system includes a screenshot file MD5 verification module, an operation log recording module, and a coordinate parameter persistent storage unit.

[0039] The screenshot file MD5 check module is configured to read the content of the screenshot file, obtain the MD5 check code of the screenshot file, and record the check code in the metadata file corresponding to the screenshot file.

[0040] Specifically, the metadata file has a.meta suffix and is named the same as the screenshot file. The screenshot file MD5 check module is configured to read the binary content of the screenshot file and obtain the MD5 check code of the screenshot file.

[0041] The operation log recording module is configured to record the key information of the operation in real time during the execution of the screenshot operation. The key information includes the operation start and end time and the operation type.

[0042] Specifically, when the operation time is not explicitly recorded, the operation start and end time can be added in the metadata file. The operation type is region screenshot by default.

[0043] The coordinate parameter persistent storage unit is configured to store the boundary coordinates of the screenshot region in the metadata file in association with the screenshot file.

[0044] All exceptions in the screenshot capturing process are captured and error information is printed through the standard error output.

[0045] Specifically, the present application captures all exceptions such as non-existent path, permission error, and screenshot failure through try-except (direct annotation method).

[0046] Further, for the two corrected coordinates, i.e., the start coordinates (x1, y1) of the screenshot region and the end coordinates (x2, y2) of the screenshot region, the sorted function (a function for sorting) is used to sort the horizontal coordinate x1 and the vertical coordinate y1 of the start coordinates of the screenshot region and the horizontal coordinate x2 and the vertical coordinate y2 of the end coordinates of the screenshot region, to ensure that x1 < x2 and y1 < y2, thereby obtaining a standard rectangular screenshot region. The width of the screenshot region is width = x2 - x1, and the height of the screenshot region is height = y2 - y1.

[0047] Further, the valid rectangular region coordinates are represented as (x1, y1, width, height). When x1 = 100, y1 = 50, width = 200, and height = 150, it represents that the top-left corner of the screenshot region is at the screen coordinate (100, 50), the rectangle extends 200 pixels to the right and 150 pixels downward, and the bottom-right corner coordinate is (x1 + width, y1 + height) = (300, 200).

[0048] The present application also includes the use of a hierarchical configuration management system to separate system-level and user-level configurations, specifically including the following content.

[0049] According to the configuration loading priority, the system-level configuration file is loaded, and then the user-level configuration file is loaded, the system-level configuration file and the user-level configuration file are merged according to a merging rule, and a merged configuration file is obtained.

[0050] The merging rule comprises: the user-level configuration covers the same configuration item in the system-level configuration, and a missing item is rolled back to a default value.

[0051] Specifically, the merging is implemented by using a configparser function (a function for merging).

[0052] The configuration item is verified, and when the value of the configuration item is not a numerical value or exceeds a preset range, the default value is rolled back.

[0053] It should be noted that the default value is the resolution of the current test environment, which can be obtained by using two independent command line tools xrandr and gsettings on Linux.

[0054] A save path of the configuration file is generated by an automatic path generation algorithm.

[0055] The automatic path generation algorithm comprises: generating a user-specific subdirectory based on a system user ID (such as os.getuid() of Linux). The screenshot file is organized in a year / month / day hierarchy. The directory is created to ensure that the path exists.

[0056] Specifically, in the present application, a user only needs to press the Ctrl key to trigger the screenshot, and only two clicks (top left and bottom right) are needed to complete the screenshot, and the system automatically processes the related information of the saved screenshot file.

[0057] S102: The operation of the intelligent area screenshot system is integrated into the automated test system, and graphical interface testing is performed.

[0058] The operation of the intelligent area screenshot system is integrated into the automated test system, and graphical interface testing is performed, which comprises the following contents.

[0059] The automated test system parses a pre-configured configuration file, and traverses each control option of the control configuration section in the configuration file. The configuration file is used to define control information in graphical interface testing.

[0060] For each control option, control information conforming to a preset format is obtained and parsed, and the control information comprises a control identifier, a resolution of a control resource map, a path, a top left coordinate, a bottom right coordinate, and an MD5 check code.

[0061] Specifically, the control resource image is a standardized visual template of the control, usually an image file in PNG / BMP format.

[0062] The parsed resolution is checked. If it is not in the valid resolution list, a resolution adjustment function is called to adjust the resolution of the control resource image.

[0063] Specifically, the valid resolution list is obtained by parsing the configuration file to obtain the resolution of each control resource image in the configuration file. The resolution is adjusted to be the same as the resolution of the current test environment.

[0064] Specifically, the resolution adjustment function can be resolution_adjust().

[0065] The parsed coordinates are checked. If the number of coordinates is not 4 or the upper-left corner coordinates are greater than or equal to the lower-right corner coordinates, a coordinate error exception is thrown.

[0066] Specifically, the parsed coordinates can be (x1, y1, x2, y2), which need to satisfy x1 < x2 and y1 < y2, otherwise a CoordinatesError exception is thrown.

[0067] The parsed and / or checked control information is stored in the control dictionary, with the dictionary key being the control identifier and the value being a dictionary including the control information.

[0068] Specifically, the control dictionary of the present application is controls[key].

[0069] Each control information in the control dictionary is traversed to determine whether the path of each control resource image exists.

[0070] If the path of the control resource image does not exist, the control positioning operation of the control resource image fails, a resolution adjustment function is called to adjust the resolution of the control resource image. According to the upper-left corner coordinates and the lower-right corner coordinates of the control resource image, a screenshot function is called to take a screenshot of the specified area, and the screenshot is saved to the configuration path, a new MD5 checksum is generated and the configuration file is updated.

[0071] If the path of the control resource image exists, an image recognition step is performed.

[0072] Specifically, the image recognition step includes the following contents.

[0073] The control resource image is read, and both the control resource image and the current interface screenshot area are converted to grayscale images and subjected to Gaussian blur preprocessing.

[0074] Specifically, the Gaussian blur preprocessing in the present application can be kernel=3x3. That is, the size of the convolution kernel kernel is 3x3.

[0075] The multi-scale template matching technology is adopted to dynamically adjust the size of the screenshot area at multiple scaling scales, and template matching is performed to obtain a matching result.

[0076] Specifically, the multiple scaling scales can be [0.9, 1.0, 1.1].

[0077] The highest similarity value in the matching result of each scale is recorded, and a preset matching threshold is compared to determine whether the recognition is successful.

[0078] Specifically, the preset matching threshold can be 0.75.

[0079] Specifically, the similarity value of the matching result in the present application can be obtained by the cv2.matchTemplate() function (template matching function) combined with the cv2.TM_CCOEFF_NORMED method (normalized correlation coefficient matching method).

[0080] If it is greater than or equal to the preset matching threshold, it is recognized successfully.

[0081] If it is less than the preset matching threshold, the recognition fails.

[0082] If the recognition is successful, graphical interface test operations are performed according to the test cases.

[0083] If the recognition fails, a resolution adjustment function is called to adjust the resolution of the control resource image. According to the top-left corner coordinates and the bottom-right corner coordinates of the control resource image, a screenshot function is called to take a new screenshot, obtain the MD5 checksum of the new screenshot, and compare it with the MD5 checksum of the control resource image in the original configuration file.

[0084] If the MD5 checksums are consistent, the user is prompted that there is a problem with the current image recognition.

[0085] If the MD5 checksums are inconsistent, the MD5 checksum of the control resource image in the configuration file is updated.

[0086] The present application integrates the operation of the intelligent area screenshot system into the automatic test system to perform graphical interface testing, and solves the problem that the automatic test needs to be re-adapted due to slight changes in the screenshot.

[0087] The present application also provides a graphical interface testing device 200, as shown in Figure 2 The device includes a construction module 201 and an integration module 202.

[0088] The construction module 201 is configured to construct the intelligent region screenshot system by using a three-layer event-driven architecture, and the intelligent region screenshot system is used for screenshot and screenshot information processing. The three-layer event-driven architecture includes an event listening layer, a coordinate processing layer and a service integration layer. The event listening layer is configured to capture a screenshot signal and an event triggered by a user operation, and record associated original boundary coordinate points. The coordinate processing layer is configured to load correction parameters from a configuration file, correct the original boundary coordinate points, and determine boundary coordinates of a screenshot region. The service integration layer is configured to perform a screenshot saving and size acquisition operation according to the determined screenshot region coordinates.

[0089] The integration module 202 is configured to integrate the operation of the intelligent region screenshot system into an automatic test system, and perform graphical interface testing.

[0090] Some of the modules in the device described in the present application can be described in the general context of computer-executable instructions, such as program modules, executed by computers. Generally, program modules include routines, programs, objects, components, data structures, classes, and the like, that perform particular tasks or implement particular abstract data types. The present application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are connected through a communication network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including storage devices.

[0091] The device or module described in the above application embodiments can be specifically implemented by a computer chip or an entity, or by a product having certain functions. For the convenience of description, the above device is described as various modules respectively described in terms of functions. In the implementation of the application embodiments, the functions of each module can be implemented in the same or multiple software and / or hardware. Of course, the modules implementing certain functions can also be implemented by multiple sub-modules or sub-units.

[0092] The methods, devices or modules described in the present application can be implemented in a computer readable program code manner, and the controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer readable medium storing computer readable program code (for example, software or firmware) executable by the (micro) processor, logic gates, switches, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable logic controllers and embedded microcontrollers. Examples of the controller include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that, in addition to implementing the controller in a pure computer readable program code manner, the same function can also be implemented by logically programming the method steps in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers. Therefore, such a controller can be considered as a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both a software module for implementing the method and a structure within the hardware component.

[0093] As shown in Figure 3 The embodiment of the present application also provides a graphical interface test server, including a memory 301 and a processor 302; the memory 301 is used for storing computer executable instructions; and the processor 302 is used for executing the computer executable instructions to realize the graphical interface test method provided by the embodiment of the present application.

[0094] The embodiment of the present application also provides a computer readable storage medium, which stores executable instructions, and when a computer executes the executable instructions, the graphical interface test method provided by the embodiment of the present application can be realized.

[0095] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and necessary hardware. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product or in the form of data migration in the implementation process. The computer software product can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the method described in the embodiments of the present application.

[0096] The various embodiments in the specification are described in progressive manner, and the same or similar parts among the various embodiments can be referred to each other, and each embodiment focuses on the difference from other embodiments. The whole or part of the present application can be used in a plurality of general or special computer system environments or configurations.

[0097] The above examples are only used to illustrate the technical solutions of the present application, and are not limited to the present application; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the present application.

Claims

1. A method of graphical interface testing, characterized by, The application relates to a three-layer event-driven architecture for constructing an intelligent region screenshot system, and a method for using the intelligent region screenshot system to take screenshots and process screenshot information. The three-layer event-driven architecture comprises an event listening layer, a coordinate processing layer and a service integration layer. The event listening layer is used for capturing screenshot signals and events triggered by user operations and recording associated original boundary coordinate points. The coordinate processing layer is used for loading correction parameters from a configuration file, correcting the original boundary coordinate points and determining boundary coordinates of a screenshot region. The service integration layer is used for performing screenshot saving and size acquisition operations according to the determined screenshot region coordinates. The operation of the intelligent region screenshot system is integrated into an automatic test system for graphical interface testing. The three-layer event-driven architecture for constructing the intelligent region screenshot system and the method for using the intelligent region screenshot system to take screenshots and process screenshot information comprise the following steps. The operation of the event listening layer comprises the following steps: constructing a global event bus based on pynput to realize a cross-process Ctrl key state synchronization mechanism; setting a keyboard and mouse event listening module to listen to Ctrl key trigger signals and mouse click events; when it is detected that the Ctrl key is pressed and a mouse click event occurs, recording original boundary coordinate points of the current mouse click and marking the coordinate points as to-be-corrected coordinates. The operation of the coordinate processing layer comprises the following steps: loading correction parameters, including display scaling ratio, height-width ratio compensation, calibration offset and DPI compensation value, from a configuration file; correcting the to-be-corrected coordinates to determine boundary coordinates of a screenshot region; wherein the boundary coordinates are two diagonal vertex coordinates of the screenshot region, including starting coordinates and ending coordinates of the screenshot region. The expression for correcting the obtained original boundary coordinate points is: ; wherein, is the corrected coordinate, is the original coordinate, is the display scaling ratio, is the calibration offset; ; wherein, is the corrected coordinate, is the screen height, is the original coordinate, is the aspect ratio compensation, is the DPI compensation value; The determination mode of the boundary coordinates of the screenshot region comprises the following steps: correcting the obtained original boundary coordinate points to obtain corrected coordinate points when the boundary coordinate points are obtained for the first time, and taking the corrected coordinate points as the starting coordinates of the screenshot region; correcting the original boundary coordinate points to obtain corrected coordinate points when the boundary coordinate points are obtained for the second time, and taking the corrected coordinate points as the ending coordinates of the screenshot region. The operation of the service integration layer comprises the following steps: performing screen region capturing and saving as a screenshot file based on the determined boundary coordinates of the screenshot region. The size of the screenshot region is obtained based on the determined boundary coordinates of the screenshot region; the size of the screenshot region comprises the height and width of the screenshot region. The saving path of the screenshot file is obtained from a configuration, a default path is used if the configuration is not provided, and it is ensured that the path exists. In the screenshot saving process, a metadata automatic generation system is started; the metadata automatic generation system comprises a screenshot file MD5 checking module, an operation log recording module and a coordinate parameter persistent storage unit. The screenshot file MD5 checking module is used for reading screenshot file content, obtaining an MD5 check code of the screenshot file and recording the check code into a metadata file corresponding to the screenshot file. The operation log recording module is used for recording key information of operations in real time in the screenshot operation execution process; the key information comprises operation start and end time and operation type. The coordinate parameter persistent storage unit is used for storing the boundary coordinates of the screenshot region in association with the screenshot file into a metadata file. Capture all exceptions in the process of taking screenshots and print error information through standard error output.

2. The graphical interface testing method of claim 1, wherein, Further comprising separating system-level and user-level configurations by using a hierarchical configuration management system, specifically comprising: Loading configuration files according to configuration loading priorities, loading a system-level configuration file first, and then loading a user-level configuration file, merging the system-level configuration file and the user-level configuration file according to merging rules to obtain a merged configuration file; The merging rules include that a user-level configuration covers a same configuration item in a system-level configuration, and a missing item reverts to a default value; Verifying configuration items, and reverting to a default value when a value of a configuration item is not a numerical value or exceeds a preset range; Generating a saving path of the configuration file through an automatic path generation algorithm.

3. The graphical interface testing method of claim 2, wherein, The operation of the intelligent area screenshot system is integrated into the automated test system for graphical interface testing, comprising: The automated test system parses a pre-configured configuration file and traverses each control option of a control configuration section in the configuration file; wherein the configuration file is used to define control information in graphical interface testing; For each control option, control information conforming to a preset format is obtained and parsed, and the control information includes a control identifier, a resolution, a path, top-left coordinates, bottom-right coordinates, and an MD5 check code of a control resource image; The parsed resolution is verified, and if it is not in a list of valid resolutions, a resolution adjustment function is called to adjust the resolution of the control resource image; The parsed coordinates are verified, and if the number of coordinates is not 4 or the top-left coordinates are greater than or equal to the bottom-right coordinates, a coordinate error exception is thrown; The parsed and / or verified control information is stored in a control dictionary, and the dictionary key is the control identifier and the value is a dictionary including the control information; Each control information in the control dictionary is traversed to determine whether the path of each control resource image exists; If the path of the control resource image does not exist, the control positioning operation of the control resource image fails, the resolution adjustment function is called to adjust the resolution of the control resource image, a screenshot function is called to take a screenshot of the specified area according to the top-left coordinates and the bottom-right coordinates of the control resource image, the screenshot is saved to the configuration path, a new MD5 check code is generated and the configuration file is updated; If the path of the control resource image exists, an image recognition step is performed; If the recognition is successful, graphical interface testing operations are performed according to a test case execution; If the recognition fails, the resolution adjustment function is called to adjust the resolution of the control resource image, a screenshot function is called to take a new screenshot according to the top-left coordinates and the bottom-right coordinates of the control resource image, an MD5 check code of the new screenshot is obtained, and the MD5 check code is compared with the MD5 check code of the control resource image in the original configuration file; If the MD5 check codes are consistent, the user is prompted that there is a problem with the current image recognition; If the MD5 check codes are inconsistent, the MD5 check code of the control resource image in the configuration file is updated.

4. The graphical interface testing method of claim 3, wherein, The image recognition step comprises: Reading the control resource image, converting the control resource image and the current interface screenshot area into grayscale images, and performing Gaussian blur preprocessing; Using a multi-scale template matching technique to dynamically adjust the size of the screenshot area at multiple scaling scales to obtain a matching result through template matching; and Record the highest similarity value in each scale matching result, compare it with a preset matching threshold, and judge whether the recognition is successful or not; If it is greater than or equal to the preset matching threshold, it is successful; If it is less than the preset matching threshold, it is failed.

5. A graphical interface testing device, characterized by It comprises: A construction module is configured to construct an intelligent region screenshot system by using a three-layer event-driven architecture, and to perform screenshot and screenshot information processing by using the intelligent region screenshot system; the three-layer event-driven architecture comprises an event listening layer, a coordinate processing layer, and a service integration layer; the event listening layer is configured to capture a screenshot signal and an event triggered by a user operation, and to record associated original boundary coordinate points; the coordinate processing layer is configured to load correction parameters from a configuration file, to correct the original boundary coordinate points, and to determine boundary coordinates of a screenshot region; and the service integration layer is configured to perform a screenshot saving and size acquisition operation according to the determined screenshot region coordinates; An integration module is configured to integrate operations of the intelligent region screenshot system into an automatic test system, and to perform graphical interface testing; The three-layer event-driven architecture comprises: Operations of the event listening layer comprise: constructing a global event bus based on pynput to realize a cross-process Ctrl key state synchronization mechanism; setting a keyboard and mouse event listening module to listen to a Ctrl key trigger signal and a mouse click event; when it is detected that the Ctrl key is pressed and the mouse click event occurs, recording original boundary coordinate points of the current mouse click, and marking them as to-be-corrected coordinates; Operations of the coordinate processing layer comprise: loading correction parameters from a configuration file, including a display scaling ratio, a height-width ratio compensation, a calibration offset, and a DPI compensation value; correcting the to-be-corrected coordinates to determine boundary coordinates of the screenshot region; wherein the boundary coordinates are two diagonal vertex coordinates of the screenshot region, including starting coordinates and ending coordinates of the screenshot region; The expression for correcting the obtained original boundary coordinate points is: ; wherein, is the corrected coordinate, is the original coordinate, is the display zoom ratio, is the calibration offset; ; wherein, is the corrected coordinate, is the screen height, is the original coordinate, is the aspect ratio compensation, is the DPI compensation value; The determination of the boundary coordinates of the screenshot region comprises: correcting the obtained original boundary coordinate points to obtain corrected coordinate points as starting coordinates of the screenshot region when the first click occurs; correcting the original boundary coordinate points to obtain corrected coordinate points as ending coordinates of the screenshot region when the second click occurs; Operations of the service integration layer comprise: performing screen region capture based on the determined boundary coordinates of the screenshot region and saving it as a screenshot file; Based on the determined boundary coordinates of the screenshot region, the size of the screenshot region is acquired; the size of the screenshot region comprises a height and a width of the screenshot region; A saving path of the screenshot file is obtained from a configuration; if no configuration is made, a default path is used, and it is ensured that the path exists; During the screenshot saving process, a metadata automatic generation system is started; the metadata automatic generation system comprises a screenshot file MD5 checking module, an operation log recording module, and a coordinate parameter persistent storage unit; The screenshot file MD5 checking module is configured to read screenshot file content, to obtain an MD5 check code of the screenshot file, and to record the check code in a metadata file corresponding to the screenshot file; The operation log recording module is configured to record key information of the operation in real time during the execution of the screenshot operation; the key information comprises operation start and end time and operation type; The coordinate parameter persistent storage unit is configured to store the boundary coordinates of the screenshot region and the screenshot file in the metadata file in association; All exceptions in the process of capturing the screenshot are captured, and error information is printed through a standard error output.

6. A graphical interface test server, characterized by comprising a memory and a processor; The memory is configured to store computer executable instructions; The processor is configured to execute the computer executable instructions to implement the method of any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer readable storage medium stores executable instructions, and a computer executing the executable instructions can implement the method of any one of claims 1-4.

Images