Method, device, storage medium and electronic device for determining interface status

Abstract

The present disclosure provides a method, device, storage medium, and electronic device for determining an interface state, relating to the field of data processing technology, and in particular, to the field of deep learning technology. A specific implementation scheme is as follows: during the execution of automated testing, a screenshot image of the display interface of the automated testing platform is obtained; the screenshot image is processed to obtain a state transition diagram, wherein the state transition diagram is used to describe at least one state parameter of the display interface; and the current interface state of the display interface is determined based on the state parameters described in the state transition diagram, wherein the current interface state includes at least one of the following: an initial state, a pop-up state, an intermediate state, and an end state.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to the field of interface automation testing technology. Background Art

[0002] Currently, when performing automated testing on the front end, the display interface jump during the use case execution mainly adopts a preset waiting time. However, insufficient waiting time can easily lead to path distortion, while too long waiting time can lead to inefficient use case execution. In addition, some display interface stability test solutions adopt offline calculation methods, which cannot provide real-time feedback on the stability of the use case execution process.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] The present disclosure provides a method, apparatus, storage medium, and electronic device for determining an interface state.

[0005] According to one aspect of the present disclosure, a method for determining an interface state is provided, comprising: obtaining a screenshot image of a display interface of an automated testing platform during automated testing; processing the screenshot image to obtain a state transition diagram, wherein the state transition diagram is used to describe at least one state parameter of the display interface; determining a current interface state of the display interface based on the state parameters described in the state transition diagram, wherein the current interface state includes at least one of the following: an initial state, a pop-up state, an intermediate state, and an end state.

[0006] According to another aspect of the present disclosure, a device for determining an interface state is provided, which is connected to an automated testing platform and includes: a screenshot module, used to obtain a screenshot image of a display interface of the automated testing platform during the execution of automated testing; a management module, connected to the above-mentioned screenshot module, used to process the above-mentioned screenshot image to obtain a state transition diagram, wherein the above-mentioned state transition diagram is used to describe at least one state parameter of the above-mentioned display interface; a strategy analysis module, connected to the above-mentioned management module, used to determine the current interface state of the above-mentioned display interface based on the above-mentioned state parameters described in the above-mentioned state transition diagram, wherein the above-mentioned current interface state includes at least one of the following: an initial state, a pop-up state, an intermediate state and an end state.

[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the methods for determining the interface status.

[0008] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the above methods for determining the interface state.

[0009] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any of the above methods for determining an interface state when executed by a processor.

[0010] According to another aspect of the present disclosure, a product for determining an interface state is provided, including the electronic device as described above.

[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present invention and do not constitute a limitation of the present invention.

[0013] Figure 1 is a schematic flow chart of steps of a method for determining an interface state according to a first embodiment of the present disclosure;

[0014] Figure 2 This is a schematic diagram of an optional process for obtaining a screenshot image of a display interface according to the first embodiment of the present disclosure;

[0015] Figure 3 is a schematic flow chart of steps of an optional method for determining an interface state according to the first embodiment of the present disclosure;

[0016] Figure 4 is a schematic flow chart of another optional method for determining an interface state according to the first embodiment of the present disclosure;

[0017] Figure 5 is a schematic diagram of an optional binary classification of a display interface based on a dense convolutional network according to the first embodiment of the present disclosure;

[0018] Figure 6 is a schematic flow chart of another optional method for determining an interface state according to the first embodiment of the present disclosure;

[0019] Figure 7 This is a schematic flow chart of an optional interface state transfer process according to the first embodiment of the present disclosure;

[0020] Figure 8is a schematic diagram comparing an optional stable interface and a predicted stable interface according to the first embodiment of the present disclosure;

[0021] Figure 9 is a structural diagram of an apparatus for determining an interface state according to a second embodiment of the present disclosure;

[0022] Figure 10 is a schematic block diagram of an optional electronic device according to the third embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] Example 1

[0026] At present, when performing automated testing on the front end, mainstream artificial intelligence AI testing platforms such as airtest, mabl, and Eggplant adopt a strategy of calling a system-level interface to wait. For example, after an action is executed, wait for 5 seconds, and then execute the next operation. If the next operation fails, it will wait for a period of time again, and the waiting time generally does not exceed 30 seconds. However, if the waiting time is insufficient, it is easy to cause path distortion, and if the waiting time is too long, it may lead to low efficiency in use case execution. At the same time, the appearance of pop-up windows is the main reason for the failure of the current test process. The pop-up window removal solution in the existing technology is mainly to passively remove the pop-up window after the page jump fails, or the user manually removes the pop-up window after discovering the pop-up window, resulting in discontinuous use case execution process, thereby affecting the use case execution efficiency.

[0027] In addition, some clients test the stability of the display interface through conventional page loading delay calculations. This involves determining the initial and stable state of page loading and then using a screenshot of the next frame to determine the stable page. This approach requires minimal computation time for a single calculation. Some display interface stability testing tools, such as the stagesepx testing tool, are time-consuming to determine the stability of the display interface. These tools primarily rely on training historical stable pages, employing machine learning methods such as support vector machines (SVMs) for binary classification to determine the stability of the current display interface. However, these methods all utilize offline calculations and cannot provide real-time feedback on the stability of the use case execution process.

[0028] In response to the above problems, the embodiments of the present disclosure provide an embodiment of a method for determining the state of an interface. The embodiments of the present disclosure require real-time online calculations, and therefore have high requirements for the speed of calculations. With the gradual improvement of real-time screenshot tools, deep learning, edge computing, and hardware image processing computing capabilities, the feasibility of online real-time page stability perception has become higher. The embodiments of the present disclosure use edge computing and NPU chips for acceleration, and use state transfer and traditional image feature extraction calculations to determine the current interface display state.

[0029] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0030] Figure 1 FIG. 1 is a flow chart of steps of a method for determining an interface state according to a first embodiment of the present disclosure. Figure 1 As shown, the method includes the following steps:

[0031] Step S102: obtaining a screenshot of a display interface of the automated testing platform during the automated testing process;

[0032] Step S104, processing the screenshot image to obtain a state transition diagram;

[0033] Step S106: determining the current interface state of the display interface based on the state parameters described in the state transition diagram.

[0034] Optionally, the above-mentioned automated testing process may include, but is not limited to, the process of loading, rendering, and jumping to a new display interface after stabilization of the display interface in functional automated testing after executing click operations, sliding operations and / or a series of pre-set operation processes (i.e., schema mode).

[0035] Optionally, the screenshots are taken during a preset waiting time for the display interface to jump when executing automated tests, and the number of frames of the screenshots taken during each preset waiting time is the same. For example, in a click scenario, a fixed waiting time (i.e., the maximum waiting time) is set to 4 seconds, denoted as timep, and the number of frames of the screenshots taken during each waiting time is n, then one screenshot frame is taken every (timep ÷ n) time.

[0036] Optionally, the state transition diagram is used to describe at least one state parameter of the display interface; the current interface state includes at least one of the following: an initial state, a pop-up state, an intermediate state, and an end state.

[0037] Optionally, the above-mentioned starting state may be, but is not limited to, the starting state of the display interface (i.e., the screenshot of the first frame) after executing a click operation, a sliding operation and / or a pre-set series of operation processes (i.e., the schema mode); the above-mentioned pop-up state is the state in which a pop-up window appears on the display interface during the execution of automated testing; the above-mentioned intermediate state may be, but is not limited to, the transit state, circling, loading plug-in, blank picture, blurred page state, etc. that appear in the display interface during the execution of automated testing; the above-mentioned end state may be, but is not limited to, the final display state in which the display interface jumps to a new display interface after loading, rendering, and stabilization during the execution of automated testing, such as a video playback state or a still display interface with pictures and text.

[0038] In an embodiment of the present disclosure, a screenshot image of the display interface of the automation test platform is obtained during the execution of the automation test; the screenshot image is processed to obtain a state transition diagram, wherein the state transition diagram is used to describe at least one state parameter of the display interface; the current interface state of the display interface is determined based on the state parameters described in the state transition diagram, wherein the current interface state includes at least one of the following: initial state, pop-up state, intermediate state and end state, thereby achieving the purpose of real-time monitoring of the jump state of the display interface during the execution of the automation test, thereby realizing the technical effect of improving the stability and test efficiency of the online automation test, and further solving the technical problem that during the execution of the front-end automation test case, the preset jump time of the display interface is inaccurate and the interface state cannot be determined, resulting in poor stability of the automation test and low test efficiency.

[0039] It should be noted that the disclosed embodiment can effectively shorten the execution time of the display interface jump action through intelligent real-time monitoring of the display interface status. For example, the action that originally waited for 5 seconds now only takes 3 seconds, which can save at least 40% of the delay and greatly improve the execution speed. In addition, due to the slower access speed caused by network status and mobile phone performance, testers do not know whether waiting for 5 seconds is enough, resulting in the next step of operation before the display interface is stable. The display interface monitoring method can wait for a longer period of time, so that the test case can be successfully executed, thereby improving overall stability.

[0040] Since the appearance of pop-up windows is a major reason for the failure of the current test process, the pop-up window removal solution in the existing technology mainly removes pop-up windows passively after the display interface fails to jump, or manually removes pop-up windows during the display interface jump, resulting in discontinuous use case execution process, thereby affecting the use case execution efficiency.

[0041] The embodiment of the present disclosure can determine the current interface state of the above-mentioned display interface based on the above-mentioned state parameters described in the above-mentioned state transition diagram, so as to realize real-time monitoring of pop-up windows, further reduce manual intervention, and be more intelligent, stable and efficient; in addition, the embodiment of the present disclosure makes full use of high-performance terminal devices through edge computing, reduces the network transmission consumption of server-side computing, and makes the entire real-time automatic judgment of the stable state of the display interface solution universal.

[0042] In an optional embodiment, obtaining a screenshot of a display interface of the automated testing platform includes:

[0043] Step S202: Using a screenshot tool to take a screenshot of the display interface of the current frame to obtain the screenshot image;

[0044] Step S204: acquiring the screenshot image in real time from the screenshot tool.

[0045] Optionally, the screenshot tool may be, but is not limited to, a screen copy tool minicap.

[0046] It should be noted that the real-time screenshot tools on mobile or PC terminals are currently well developed. For example, minicap can achieve a screenshot rate of 30 frames per second, continuously perform high-speed screenshots, and is stable and efficient and suitable for multiple platforms. The above-mentioned screenshot tools can achieve the purpose of obtaining screenshot images in real time, and then realize real-time monitoring of the display interface status, effectively improving the technical effect of executing the display interface jump action.

[0047] As an optional embodiment, Figure 2 FIG. 1 is a flow chart of an optional step of obtaining a screenshot image of a display interface according to the first embodiment of the present disclosure. Figure 2As shown, the screenshot image acquisition scheme includes: a management thread manager, a screenshot thread snapshot-taker, a strategy thread strategy, and a strategy analysis thread analyze, wherein the management thread is used to schedule the real-time screenshot tool and select the most recent frame of the image to return to the strategy thread; the screenshot thread is used to return the screenshot image of the 0th frame to the management thread, and the strategy thread is used to send a screenshot image acquisition request, that is, the request sent requires the screenshot image of the Nth frame; the strategy analysis thread is used to analyze whether the current display interface state is a thread of the focus state. The specific implementation process is: the screenshot thread returns the screenshot image of the 0th frame (that is, the frame number of the initial display interface) to the management thread, and the management thread continuously sends the screenshot image to the strategy thread in real time, and the strategy analysis thread analyzes and feeds back to the management thread in real time the frame number of the next frame, as well as information such as whether the current display interface state is a thread of the focus state.

[0048] In an optional embodiment, the processing of the screenshot image to obtain a state transition diagram includes:

[0049] Step S302: Process the screenshot image using a state transition algorithm to obtain the state transition diagram.

[0050] Optionally, the state transition diagram is used to describe at least one state parameter of the display interface; the current interface state includes at least one of the following: an initial state, a pop-up state, an intermediate state, and an end state.

[0051] In the embodiment of the present disclosure, the state transition diagram is obtained by processing the screenshot image using the state transition algorithm, so as to achieve the purpose of obtaining the current interface state in a timely and accurate manner.

[0052] In an optional embodiment, the determining of the current interface state of the display interface based on the state parameters described in the state transition diagram includes:

[0053] Step S402: determining whether the display interface has changed compared to the initial frame interface based on the state parameters described in the state transition diagram;

[0054] Step S404: If the display interface has not changed compared to the initial frame interface, then determining that the current interface state is the initial state.

[0055] Optionally, an interface similarity value is determined based on the above-mentioned state parameters, wherein the above-mentioned interface similarity value is used to judge whether the display interface has changed compared to the initial frame interface. The above-mentioned interface similarity value can be judged by, but not limited to, hash fingerprint, SSIM and other algorithms, and when the interface similarity value is higher than a preset value, it is considered that the above-mentioned display interface has not changed compared to the above-mentioned initial frame interface, and the above-mentioned current interface state is determined to be the above-mentioned initial state.

[0056] Optionally, the initial state may be, but is not limited to, the initial state of the display interface (ie, the screenshot of the first frame) after executing a click operation, a sliding operation and / or a pre-set series of operation processes (ie, schema mode).

[0057] It should be noted that the above method of determining the interface state determines whether the above current interface state is the initial state by judging whether the current interface state is consistent with the initial frame interface (i.e., the first frame screenshot image), thereby achieving the purpose of quickly and accurately judging whether the above current interface state is the initial state.

[0058] As an optional embodiment, Figure 3 FIG. 1 is a flow chart of steps of an optional method for determining an interface state according to the first embodiment of the present disclosure. Figure 3 As shown, the above-mentioned state parameters described in the above-mentioned state transition diagram are used to determine the current interface state of the above-mentioned display interface, including:

[0059] Step S502: determining whether the display interface has changed compared to the initial frame interface based on the state parameters described in the state transition diagram;

[0060] Step S504: if the display interface changes compared to the initial frame interface, detecting the interface blank rate of the display interface;

[0061] Step S506 , in response to the interface blank rate being greater than or equal to a preset blank rate threshold, determining that the current interface state is the intermediate state, where the intermediate state is used to indicate that the display interface is in an unstable display state.

[0062] Optionally, an interface similarity value is determined based on the above-mentioned state parameters, wherein the above-mentioned interface similarity value is used to judge whether the display interface has changed compared with the initial frame interface. The above-mentioned interface similarity value can be judged by, but not limited to, hash fingerprint, SSIM and other algorithms. When the interface similarity value is lower than a preset value, it is considered that the above-mentioned display interface has changed compared with the above-mentioned initial frame interface, and further judged whether the above-mentioned current interface state is the above-mentioned intermediate state based on the interface blank rate.

[0063] Optionally, before calculating the interface blank rate, the current display interface can be preprocessed by an image recognition algorithm. For example, the edges of the blurred image are first filtered by an edge detection method, and some other algorithms, such as pixel-level scanning technology, are used to remove the skeleton screen (that is, determine whether there are extra horizontal lines in the current interface, and remove them if so), and calculate the interface blank rate based on the processed display interface.

[0064] It should be noted that the above-mentioned intermediate state is during the display interface rendering process (the front-end display interface is generally composed of various display interface components, and rendering is the process of adding content to the components). In the intermediate state, some or all components will be in a blank state. The interface blank rate can be expressed as the percentage of the blank area in the current display interface to the total area of ​​the display interface. When the above-mentioned interface blank rate is higher than a certain threshold, such as 80%, it is determined that the current interface state is an intermediate state. The above-mentioned method of judging whether the above-mentioned current interface state is the above-mentioned intermediate state based on the interface blank rate can quickly identify the intermediate state, thereby effectively shortening the execution time of the display interface jump action and improving the execution speed.

[0065] Optionally, the above intermediate state may be, but is not limited to, a transit state, a spinning circle, a loading plug-in, a blank image, a blurred page state, etc. that appears in the display interface during the execution of the automated test.

[0066] As an optional embodiment, Figure 4 is a flowchart of another optional method for determining the interface state according to the first embodiment of the present disclosure. Figure 4 As shown, the above-mentioned state parameters described in the above-mentioned state transition diagram are used to determine the current interface state of the above-mentioned display interface, including:

[0067] Step S602: determining whether the display interface has changed compared to the initial frame interface based on the state parameters described in the state transition diagram;

[0068] Step S604: if the display interface has changed compared to the initial frame interface, detecting the interface blank rate of the display interface;

[0069] Step S606, in response to the interface blank rate being less than a preset blank rate threshold, the current interface state is determined to be the terminal state or the pop-up state, wherein the terminal state is used to characterize that the display interface is in a stable display state, and the pop-up state is used to characterize the presence of a pop-up control on the display interface.

[0070] Optionally, an interface similarity value is determined based on the above-mentioned state parameters, wherein the above-mentioned interface similarity value is used to determine whether the display interface has changed compared to the initial frame interface. The above-mentioned interface similarity value can be judged by, but not limited to, hash fingerprint, SSIM and other algorithms. When the interface similarity value is lower than a preset value, it is determined that the above-mentioned display interface has changed compared to the above-mentioned initial frame interface, and when the interface blank rate is lower than a preset blank rate threshold, it is determined that the above-mentioned current interface state is the above-mentioned end state or the above-mentioned pop-up window state.

[0071] Optionally, based on deep learning and edge computing, it is determined in real time whether there is a pop-up control on the above display interface, that is, whether the above current interface state is a pop-up state.

[0072] It should be noted that general deep learning capabilities are deployed on the server side, and edge computing is a capability to perform calculations on the terminal close to data acquisition. Its advantage is that it can make full use of nearby device resources. With the increasing popularity of a number of high-performance devices, such as image computing chips (NPU chips, APU chips, XPU chips), and PC-side GPU series, the feasibility of real-time image computing has become higher. By judging in real time whether there are pop-up controls on the above display interface based on deep learning and edge computing, it is possible to reduce network transmission consumption and effectively shorten the display interface jump delay, thereby reducing the sense of interruption in the online automated testing process, reducing manual intervention, and improving the stability and test efficiency of online automated testing.

[0073] Optionally, the display interface is classified into two categories through a network structure (such as a dense convolutional network densenet, a lightweight network mobilenet, etc.), that is, the display interface is divided into a pop-up state and a non-pop-up state according to the pop-up display situation. Figure 5 A schematic diagram of an optional binary classification of a display interface based on a dense convolutional network is shown. As shown in the figure, the display interface is trained based on a dense convolutional network, and the current interface state is determined to be a pop-up state based on an inference framework. Optionally, the inference framework may be, but is not limited to, a paddlelite inference framework or a tensorlite inference framework, wherein the paddlelite inference framework is suitable for mobile terminal devices, and the tensorlite inference framework is suitable for PCs. It should be noted that the network structure can achieve a classification and recognition speed of 30ms on an embedded neural network processing device NPU, and the image processor GPU on the PC side can also be faster, achieving a near real-time classification effect.

[0074] As an optional embodiment, Figure 6 is a flowchart of another optional method for determining the interface state according to the first embodiment of the present disclosure. Figure 6 As shown, the above-mentioned state parameters described in the above-mentioned state transition diagram are used to determine whether the above-mentioned display interface has changed compared to the initial frame interface, including:

[0075] Step S702, obtaining the first state parameter of the display interface described in the state transition diagram and the second state parameter corresponding to the initial frame interface;

[0076] Step S704: calculating a similarity value between the display interface and the initial frame interface based on the first state parameter and the second state parameter;

[0077] Step S706 , determining whether the display interface has changed compared to the initial frame interface based on the similarity value.

[0078] Optionally, the above interface similarity value can be determined by, but is not limited to, hash fingerprint, SSIM and other algorithms, and when the interface similarity value is higher than a preset value, it is considered that the above display interface has not changed compared to the above initial frame interface.

[0079] In an embodiment of the present disclosure, by obtaining the first state parameter of the above-mentioned display interface described in the above-mentioned state transition diagram and the second state parameter corresponding to the above-mentioned initial frame interface; based on the above-mentioned first state parameter and the above-mentioned second state parameter, calculating the similarity value between the above-mentioned display interface and the above-mentioned initial frame interface; and determining whether the above-mentioned display interface has changed compared with the above-mentioned initial frame interface based on the above-mentioned similarity value, the purpose of timely obtaining and feedback of the display state information of the display interface can be achieved, thereby realizing the technical effect of improving the efficiency of online automated testing.

[0080] As an optional embodiment, Figure 7 is a flowchart of an optional interface state transfer process according to the first embodiment of the present disclosure, such as Figure 7 As shown, the above-mentioned sampling frame number n=4 is determined, and the waiting time (i.e., the maximum waiting time) is 4 seconds, which is recorded as timep. The above-mentioned interface state transfer process includes: passing the initial frame (i.e., the first frame) interface, and for the above-mentioned initial frame interface, it is recorded as the initialization state status=-1; determining that the state when the time offset is 0 is the initial state, which is recorded as status=0, and executing the return operation. After determining that the current strategy result is normal and the current loop ends, requesting access to the time offset nextframe of the next frame, wherein the time offset of the next frame nextframe=nextframe+timep / n.

[0081] The second frame display interface is passed in, and the similarity value between the above-mentioned second frame display interface and the initial frame interface is calculated based on the state parameters. If the above-mentioned similarity value is higher than the preset value, the current interface display state is kept unchanged and is still the above-mentioned initial state status=0; if the above-mentioned similarity value is lower than the preset value, it is determined whether the current interface display state is an intermediate state. If so, the above-mentioned current interface display state is updated to the intermediate state status=1, and a return operation is performed. After determining that the current strategy result is normal and the current loop is completed, a request is made to access the time offset nextframe of the next frame, where the time offset of the next frame nextframe=nextframe+timep / n; otherwise, the above-mentioned current interface display state is updated to the pre-end state status=2, and a return operation is performed. After determining that the current strategy result is normal and the current loop is completed, a request is made to access the time offset nextframe of the next frame, where the time offset of the next frame nextframe=nextframe+timep / 2n.

[0082] The third frame display interface is passed in to determine whether the current interface display state is status = 1. If so, the intermediate state continues for 2 times, and the current interface display state is changed to status = 2; if the current interface display state is status = 2, the current interface display state is changed to the terminal state status = 3, and the return operation is performed. After determining that the current strategy result is normal and the current cycle is completed, the time offset of the current stable frame is returned, and the interface state transfer process ends.

[0083] Otherwise, the fourth frame display interface is passed in, and the above interface display status judgment operation is repeated until the number of frames of the current display interface exceeds 6 frames. The return operation is executed to determine that the current strategy result fails and the current loop ends.

[0084] It should be noted that the above-mentioned interface state transition process automatically stops after the number of frames of the display interface exceeds 6 frames, which is 2 frames more than the sampling frame number n = 4. This effectively solves the path distortion problem caused by too short a waiting time and the inefficiency problem caused by too long a waiting time. After the current interface display state is status = 2, the time offset nextframe changes from nextframe + timep / n to nextframe + timep / 2n, and the sampling interval is shortened to half of the original, thereby reducing waiting time and accelerating the entry into the terminal state. At the same time, the accuracy of the current interface display state judgment is recorded in real time. If the accuracy falls below the threshold, the sampling interval is appropriately extended.

[0085] Optionally, since the calculation based on interface features (such as the similarity value calculated based on state parameters) cannot be completely accurate, there is a certain probability that it cannot be accurately judged when encountering small target objects, for example, Figure 8 In the stable interface (left) and the predicted stable interface (right), the key button (the button marked with three dots) on the predicted stable page in the right image is not recognized, preventing the user from performing the relevant click operations and causing path distortion. This small target problem can be solved by adding user templates, that is, using template matching to determine whether the target state is stable. The user templates can be obtained, but are not limited to, using the visual computing library Templematch and / or deep learning algorithms.

[0086] It should be noted that the optional or preferred implementations of this embodiment can be found in the relevant descriptions of the above-mentioned method embodiment for determining the interface state, which will not be repeated here. In the technical solution disclosed herein, the acquisition, storage, and application of user personal information involved are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0087] Example 2

[0088] According to an embodiment of the present disclosure, there is also provided an embodiment of a device for implementing the above method for determining the interface state. Figure 9 is a structural diagram of a device for determining an interface state according to a second embodiment of the present disclosure, such as Figure 9 As shown, the above-mentioned device for determining the interface status includes: a screenshot module 20, a management module 22, and a strategy analysis module 24, wherein:

[0089] The above-mentioned screenshot module 20 is used to obtain a screenshot image of the display interface of the automated testing platform during the execution of automated testing; the above-mentioned management module 22 is connected to the above-mentioned screenshot module, and is used to process the above-mentioned screenshot image to obtain a state transition diagram, wherein the above-mentioned state transition diagram is used to describe at least one state parameter of the above-mentioned display interface; the above-mentioned strategy analysis module 24 is connected to the above-mentioned management module, and is used to determine the current interface state of the above-mentioned display interface based on the above-mentioned state parameters described in the above-mentioned state transition diagram, wherein the above-mentioned current interface state includes at least one of the following: initial state, pop-up state, intermediate state and end state.

[0090] In the embodiment of the present disclosure, the screenshot module 20 is used to obtain a screenshot image of the display interface of the automated testing platform during the execution of automated testing; the management module 22 is connected to the screenshot module and is used to process the screenshot image to obtain a state transition diagram, wherein the state transition diagram is used to describe at least one state parameter of the display interface; the strategy analysis module 24 is connected to the management module and is used to determine the current interface state of the display interface based on the state parameters described in the state transition diagram, wherein the current interface state includes at least one of the following: initial state, pop-up state, intermediate state and end state, so as to monitor the jump state of the display interface in real time, thereby achieving the technical effect of improving the stability and test efficiency of online automated testing, and further solving the technical problem that during the execution of front-end automated test cases, the preset jump time of the display interface is inaccurate and the interface state cannot be determined, resulting in poor stability of the automated test and low test efficiency.

[0091] It should be noted that the above modules can be implemented by software or hardware. For example, for the latter, it can be implemented in the following ways: the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0092] It should be noted that the screenshot module 20, management module 22, and strategy analysis module 24 correspond to steps S102 to S106 in Example 1. The examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules, as part of the device, can be run in a computer terminal.

[0093] It should be noted that the optional or preferred implementation of this embodiment can be found in the relevant description of Example 1 and will not be repeated here. In the technical solution disclosed herein, the acquisition, storage, and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0094] Example 3

[0095] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0096] Figure 10A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0097] like Figure 10 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the device 800 can also be stored in the RAM 803. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0098] Various components in device 800 are connected to I / O interface 805, including an input unit 806, such as a keyboard, mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, optical disk, etc.; and a communication unit 809, such as a network card, modem, wireless communication transceiver, etc. The communication unit 809 allows device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0099] The computing unit 801 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the method of obtaining a screenshot image of the display interface of the automated testing platform during the execution of the automated test. For example, in some embodiments, the method of obtaining a screenshot image of the display interface of the automated testing platform during the execution of the automated test can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of obtaining a screenshot image of the display interface of the automated testing platform during the execution of the automated test can be performed. Alternatively, in other embodiments, the computing unit 801 may be configured in any other appropriate manner (eg, by means of firmware) to execute the method to obtain a screenshot image of the display interface of the automated testing platform during the execution of the automated testing.

[0100] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0101] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0102] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0103] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0104] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0105] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0106] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0107] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for determining an interface state, comprising: During the execution of the automated test, a screenshot of the display interface of the automated test platform is obtained; Processing the screenshot image to obtain a state transition diagram, wherein the state transition diagram is used to describe at least one state parameter of the display interface, and the state parameter is used to determine an interface similarity value; Determining a current interface state of the display interface based on the state parameters described in the state transition diagram, wherein the current interface state includes at least one of the following: an initial state, a pop-up state, an intermediate state, and an end state; Among them, the determining of the current interface state of the display interface based on the state parameters described in the state transition diagram includes: determining whether the display interface has changed compared with the initial frame interface based on the state parameters described in the state transition diagram; if the display interface has changed compared with the initial frame interface, detecting the interface blank rate of the display interface; in response to the interface blank rate being less than a preset blank rate threshold, determining that the current interface state is the terminal state or the pop-up window state, wherein the terminal state is used to characterize that the display interface is in a stable display state, and the pop-up window state is used to characterize the presence of a pop-up window control on the display interface, and whether the pop-up window control exists is determined by performing real-time judgment on the display interface using deep learning and edge computing.

2. The method according to claim 1, wherein The determining the current interface state of the display interface based on the state parameters described in the state transition diagram includes: Determining whether the display interface has changed compared to the initial frame interface based on the state parameters described in the state transition diagram; If the display interface has not changed compared to the initial frame interface, the current interface state is determined to be the initial state.

3. The method according to claim 1, wherein The determining the current interface state of the display interface based on the state parameters described in the state transition diagram includes: Determining whether the display interface has changed compared to the initial frame interface based on the state parameters described in the state transition diagram; If the display interface changes compared to the initial frame interface, detecting the interface blank rate of the display interface; In response to the interface blank rate being greater than or equal to a preset blank rate threshold, the current interface state is determined to be the intermediate state, and the intermediate state is used to represent that the display interface is in an unstable display state.

4. The method according to any one of claims 2 to 3, wherein: The determining, based on the state parameters described in the state transition diagram, whether the display interface has changed compared to the initial frame interface includes: Acquire a first state parameter of the display interface described by the state transition diagram and a second state parameter corresponding to the initial frame interface; Calculating a similarity value between the display interface and the initial frame interface based on the first state parameter and the second state parameter; Determine whether the display interface has changed compared to the initial frame interface according to the similarity value.

5. The method according to claim 1, wherein The processing of the screenshot image to obtain a state transition diagram includes: The screenshot image is processed using a state transition algorithm to obtain the state transition diagram.

6. The method according to claim 1, wherein The obtaining of a screenshot of the display interface of the automated testing platform includes: Using a screenshot tool to take a screenshot of the display interface of the current frame to obtain the screenshot image; The screenshot image is acquired in real time from the screenshot tool.

7. A device for determining interface status, connected to an automated testing platform, comprising: A screenshot module is used to obtain a screenshot image of the display interface of the automated testing platform during the execution of the automated testing; a management module connected to the screenshot module, configured to process the screenshot image to obtain a state transition diagram, wherein the state transition diagram is used to describe at least one state parameter of the display interface, and the state parameter is used to determine an interface similarity value; a strategy analysis module, connected to the management module, for determining a current interface state of the display interface based on the state parameters described in the state transition diagram, wherein the current interface state includes at least one of the following: an initial state, a pop-up state, an intermediate state, and an end state; Among them, the strategy analysis module is also used to: determine whether the display interface has changed compared to the initial frame interface based on the state parameters described in the state transition diagram; if the display interface has changed compared to the initial frame interface, detect the interface blank rate of the display interface; in response to the interface blank rate being less than a preset blank rate threshold, determine that the current interface state is the terminal state or the pop-up window state, wherein the terminal state is used to characterize that the display interface is in a stable display state, and the pop-up window state is used to characterize the presence of a pop-up window control on the display interface, and whether the pop-up window control exists is determined by using deep learning and edge computing to perform real-time judgment on the display interface.

8. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method for determining the interface state according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the method for determining the interface state according to any one of claims 1 to 6. 10 . A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method for determining an interface state according to claim 1 .

11. A product for determining an interface state, comprising the electronic device according to claim 8.

Images