Method and device for automatically determining interactive GUI elements to be interacted with in a GUI

Abstract

A computer-implemented method, device, and non-transitory computer-readable storage medium for automatically determining an interactive GUI element to be interacted with in a graphical user interface (GUI). The method includes: a processor detecting one or more candidate interactive GUI elements in the GUI based on multiple algorithms; the processor determining a likelihood indicator for each of the one or more candidate interactive GUI elements, the likelihood indicator indicating the likelihood that the candidate interactive GUI element associated with the likelihood indicator is the interactive GUI element to be interacted with; and the processor determining the interactive GUI element to be interacted with from the one or more candidate interactive GUI elements based on the likelihood indicator.

Description

[0001] Cross-references

[0002] Several references, which may include patents, patent applications, and various publications, are cited and discussed in the description of the present disclosure. Citation and / or discussion of such references is provided solely to clarify the description of the present disclosure and is not an admission that any such reference is "prior art" to the disclosure described herein. All references cited and discussed in this specification are incorporated herein by reference in their entirety to the same extent as if each reference were incorporated by reference individually. Technical Field

[0003] The present disclosure relates to the field of automated control, and more particularly to a computer-implemented method, device, and non-transitory computer-readable storage medium for automatically determining user-interactive malware. Background Art

[0004] With the increasing popularity of electronic devices such as computers and mobile phones, they have become an indispensable part of our daily lives. However, malware has become a threat to such electronic devices.

[0005] Malware (a common name for malicious software) is any software that is intentionally designed to cause damage to a computer, server, client, or computer network. Malware is implanted or introduced into a target computer or mobile phone in a way that causes damage and can take the form of executable code, scripts, active content, and other software. This code is described as computer viruses, worms, Trojan horses, ransomware, spyware, adware, or scareware, among others. Malware has malicious intent and acts against the interests of the computer user, so it does not include software that causes accidental damage due to some defect (which is usually described as a software bug).

[0006] The study of malware behavior can be used to develop anti-malware solutions to prevent malware attacks. Sandbox technology is commonly used to execute untrusted programs in a strictly controlled environment without harming the host device [2]. For large-scale malware behavior research, automated techniques without user interaction have been developed to improve the efficiency of sandboxes [3]. However, malware is constantly evolving and involves complex graphical user interfaces (GUIs) before exhibiting malicious behavior.

[0007] Therefore, there exists an unresolved need in the art to address the above-mentioned deficiencies and inadequacies. Summary of the invention

[0008] The present disclosure provides a computer-implemented method, apparatus, and non-transitory computer-readable storage medium for automatically determining interactive GUI elements to be interacted with in a graphical user interface (GUI).

[0009] According to one aspect, a computer-implemented method is provided for automatically determining an interactive GUI element to be interacted with in a graphical user interface (GUI). The method includes: a processor detecting one or more candidate interactive GUI elements in the GUI based on multiple algorithms; the processor determining a likelihood indicator for each of the one or more candidate interactive GUI elements, the likelihood indicator indicating the likelihood that the candidate interactive GUI element associated with the likelihood indicator is the interactive GUI element to be interacted with; and the processor determining the interactive GUI element to be interacted with from the one or more candidate interactive GUI elements based on the likelihood indicator.

[0010] In some embodiments, the plurality of algorithms include at least one of: an algorithm based on an application programming interface (API); an algorithm based on optical character recognition (OCR); and an algorithm based on entropy.

[0011] In some embodiments, the step of a processor detecting one or more candidate interactive GUI elements in a GUI based on multiple algorithms includes: the processor extracts text from each of the one or more interactive GUI elements in the GUI by using an API of the GUI; the processor determines a first algorithm-specific likelihood for each of the one or more interactive GUI elements based on the extracted text; for each interactive GUI element whose calculated first algorithm-specific likelihood is greater than a first algorithm-specific threshold, the processor determines the interactive GUI element as a candidate interactive GUI element.

[0012] In some embodiments, the step of the processor determining a first algorithm-specific likelihood for each of the one or more interactive GUI elements based on the extracted text includes: the processor comparing the extracted text with one or more texts in a predefined set of texts; in response to the comparison result indicating that the extracted text matches a text, the processor determines the likelihood corresponding to the text in the predefined set of texts as the first algorithm-specific likelihood of the corresponding interactive GUI element.

[0013] In some embodiments, the processor determining a likelihood indicator for each of the one or more candidate interactive GUI elements includes the processor determining the likelihood indicator by weighting the first algorithm-specific likelihood with a first algorithm-specific weight.

[0014] In some embodiments, the step of the processor detecting one or more candidate interactive GUI elements in the GUI based on multiple algorithms includes: the processor detects one or more text blocks in the GUI by using an OCR-based algorithm; the processor calculates the density and / or the second algorithm-specific likelihood for each of the one or more text blocks based on the text in the corresponding text block; for each text block whose density is greater than the second algorithm-specific threshold and / or the second algorithm-specific likelihood is greater than the first algorithm-specific threshold, the processor determines the interactive GUI element associated with the text block as a candidate interactive GUI element.

[0015] In some embodiments, the step of the processor calculating the density for each of the one or more text blocks based on the text in the corresponding text block includes: the processor determining the number of target words and the number of all words in the text in the corresponding text block; and the processor calculating the density by dividing the number of all words by the number of target words.

[0016] In some embodiments, the processor determining a likelihood indicator for each of the one or more candidate interactive GUI elements includes the processor determining the likelihood indicator by weighting the second algorithm-specific likelihood with a third algorithm-specific weight.

[0017] In some embodiments, the step of the processor detecting one or more candidate interactive GUI elements in the GUI based on multiple algorithms includes: the processor obtaining a GUI image corresponding to the GUI; the processor dividing the GUI image into multiple blocks; the processor determining the entropy for each of the multiple blocks by using an entropy-based algorithm; and for each block whose entropy is greater than a threshold specified by a third algorithm, the processor determines the interactive GUI element associated with the block as a candidate interactive GUI element.

[0018] In some embodiments, the step of the processor determining entropy for each of the multiple blocks by using an entropy-based algorithm includes: the processor determining an entropy value for each of the multiple blocks based on the intensity and / or edge orientation of pixels in the corresponding block in the GUI image.

[0019] In some embodiments, the step of the processor determining a likelihood indicator for each of the one or more candidate interactive GUI elements includes the processor determining the likelihood indicator by weighting the entropy using a third algorithm-specific weight.

[0020] In some embodiments, the step of the processor determining an interactive GUI element to be interacted with from one or more candidate interactive GUI elements based on a likelihood indicator includes: the processor sorting one or more candidate interactive GUI elements according to the likelihood indicator; and the processor determining the candidate interactive GUI element with the highest likelihood indicator as the interactive GUI element to be interacted with.

[0021] In some embodiments, the method further includes: the processor comparing the information related to the determined interactive GUI element to be interacted with historical information previously stored in a database.

[0022] In some embodiments, the method also includes: in response to determining that the information related to the determined interactive GUI element to be interacted with does not match the historical information previously stored in the database, the processor interacts with the determined interactive GUI element to be interacted with by simulating user operations corresponding to the type of interactive GUI element to be interacted with.

[0023] In some embodiments, the method also includes: the processor determining whether the state of the GUI changes after the interaction; and in response to determining that the state of the GUI is changed after the interaction, the processor records information related to the determined interactive GUI element in a database as historical information.

[0024] In some embodiments, the method also includes: in response to determining that information related to the determined interactive GUI element to be interacted matches historical information previously stored in a database, the processor determines another interactive GUI element to be interacted from one or more candidate interactive GUI elements based on the likelihood indicator.

[0025] In some embodiments, the information related to the determined interactive GUI element to be interacted includes at least one of: an identifier of the GUI; a title of the determined interactive GUI element to be interacted with; a position of the determined interactive GUI element to be interacted with; a size of the determined interactive GUI element to be interacted with; and a position at which the interaction is to occur.

[0026] In some embodiments, the application corresponding to the GUI is executed in a sandbox.

[0027] According to another aspect, a device is provided. The device includes a processor; a memory storing instructions, which, when executed by the processor, causes the processor to: detect one or more candidate interactive GUI elements in a GUI based on multiple algorithms; determine a likelihood indicator for each of the one or more candidate interactive GUI elements, the likelihood indicator indicating the likelihood that the candidate interactive GUI element associated with the likelihood indicator is an interactive GUI element to be interacted with; and determine the interactive GUI element to be interacted with from the one or more candidate interactive GUI elements based on the likelihood indicator.

[0028] According to yet another aspect, a non-transitory computer-readable storage medium storing instructions is provided, which instructions, when executed by a processor, cause the processor to perform the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings illustrate one or more embodiments of the present disclosure and together with the description are used to explain the principles of the present disclosure. In all drawings, the same reference numerals are used to refer to the same or similar elements of the embodiments as much as possible.

[0030] Figure 1A is a diagram showing an exemplary resource graph without a sandbox.

[0031] Figure 1B is a diagram showing an exemplary resource graph with sandboxing.

[0032] Figure 2 is a diagram showing an exemplary Windows dialog box.

[0033] Figure 3 is a flowchart illustrating an exemplary method of automatically determining interactive GUI elements to be interacted with in a graphical user interface (GUI) according to an embodiment of the present disclosure.

[0034] Figure 4 is a flowchart illustrating an exemplary method of determining interactive GUI elements by using an API-based algorithm according to an embodiment of the present disclosure.

[0035] Figure 5 is a diagram illustrating an exemplary GUI to be processed by an API-based algorithm according to an embodiment of the present disclosure.

[0036] Figure 6 is a flow chart illustrating an exemplary method of determining interactive GUI elements by using a density-based algorithm according to an embodiment of the present disclosure.

[0037] Figure 7 is a diagram illustrating another exemplary GUI to be processed by a density-based algorithm according to an embodiment of the present disclosure.

[0038] Figure 8 is a flow chart illustrating an exemplary method of determining interactive GUI elements by using an entropy-based algorithm according to an embodiment of the present disclosure.

[0039] Fig. 9 is a diagram illustrating an exemplary GUI to be processed by an entropy-based algorithm according to an embodiment of the present disclosure.

[0040] Fig.10 The present invention is shown in Figure 3 Flowchart of an exemplary implementation of some steps of the method shown in FIG.

[0041] Fig.11 is a block diagram illustrating an exemplary arrangement that may be used in an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] The present disclosure is described in more detail in the following examples, which are intended to be illustrative only, since many modifications and variations therein will be clear to those skilled in the art in light of the teachings of the present disclosure. Various embodiments of the present disclosure are now described in detail. With reference to the accompanying drawings, in all figures, the same reference numerals represent the same components. As used in the specification herein and throughout the claims that follow, the meanings of "a", "an" and "the" include plural forms unless the context clearly indicates otherwise. Moreover, for the convenience of the reader, titles or subtitles may be used in the specification, which will not affect the scope of the present disclosure. In addition, some of the terms used in this specification are defined in more detail below.

[0043] In the context of the present disclosure and in the specific context in which each term is used, the terms used in this specification generally have the ordinary meaning of the art. Certain terms used to describe the present disclosure are discussed below or elsewhere in the specification to provide additional guidance to practitioners about the description of the present disclosure. It should be understood that the same thing can be described in more than one way. Therefore, alternative language and synonyms can be used for any one or more terms discussed herein, and whether the terms are elaborated or discussed in this article does not have any special meaning. Synonyms for certain terms are provided. The recording of one or more synonyms does not exclude the use of other synonyms. The examples used anywhere in this specification (including examples of any terms discussed herein) are only illustrative and in no way limit the scope and meaning of the present disclosure or any exemplary term. Similarly, the present disclosure is not limited to the various embodiments given in this specification.

[0044] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the present disclosure belongs. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted in an idealized manner or overly formal sense unless explicitly defined herein.

[0045] As used herein, "approximately," "about," "roughly," or "approximately" shall generally mean within 20 percent, within 10 percent, and / or within 5 percent of a given value or range. The values ​​given herein are approximate, meaning that if not explicitly stated, the term "approximately," "about," "roughly," or "approximately" can be inferred.

[0046] As used herein, the term "plurality" means two or more.

[0047] As used herein, the terms "comprising," "inclduing," "carrying," "having," "including," "involving," etc. should be understood as open-ended, meaning including but not limited to.

[0048] As used herein, the statement at least one of A, B, and C should be construed to mean a logical (A or B or C) using a non-exclusive logical "or." It should be appreciated that one or more steps within a method may be performed in a different order (or simultaneously) without altering the principles of the present disclosure. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0049] As used herein, the term "module" may refer to, be a part of, or include: an application specific integrated circuit (ASIC); an electronic circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor (a shared processor, a dedicated processor, or a processor group) that executes code; other suitable hardware components that provide the above functions; or a combination of some or all of the above, such as in a system on a chip. The term "module" may include a memory (a shared memory, a dedicated memory, or a memory group) that stores code executed by a processor.

[0050] As used herein, the term "code" may include software, firmware and / or microcode, and may refer to a program, a routine, a function, a class and / or an object. As used above, the term "shared" refers to the ability to use a single (shared) processor to execute some or all of the code from multiple modules. In addition, some or all of the code from multiple modules may be stored by a single (shared) memory. As used above, the term "group" refers to the ability to use a group of processors to execute some or all of the code from a single module. In addition, a group of memories may be used to store some or all of the code from a single module.

[0051] As used herein, the term "interface" generally refers to a communication tool or device used to perform data communication between components at an interaction point between components. Typically, an interface can be applied at both the hardware and software levels and can be a unidirectional or bidirectional interface. Examples of physical hardware interfaces can include electrical connectors, buses, ports, cables, terminals, and other I / O devices or components. The components that communicate with the interface can be, for example, multiple components or peripheral devices of a computer system.

[0052] The present disclosure relates to computer systems. As depicted in the figure, computer components may include physical hardware components (sometimes shown as solid-line boxes) and virtual software components (sometimes shown as dashed-line boxes). It will be understood by those of ordinary skill in the art that, unless otherwise indicated, these computer components may be implemented in the form of software, firmware, or hardware components, or a combination thereof (but not limited thereto).

[0053] The devices, systems, and methods described herein may be implemented by one or more computer programs executed by a processor. The computer program includes processor executable instructions that are stored on a non-transitory tangible computer readable storage medium. The computer program may also include stored data. Non-limiting examples of non-transitory tangible computer readable media are non-volatile memory, magnetic storage, and optical storage.

[0054] The present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which embodiments of the present disclosure are shown. However, the present disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; the present disclosure is not limited to the disclosed embodiments. Rather, these embodiments are provided to make the present disclosure thorough and complete and will fully convey the scope of the present disclosure to those skilled in the art.

[0055] Before giving a detailed description of some embodiments of the present disclosure, some terms used herein will be briefly explained.

[0056] Optical Character Recognition (OCR): OCR refers to the mechanical or electronic conversion of images of typed, handwritten, or printed text into machine-encoded text, whether they come from scanned documents, photographs of documents, photographs of scenes (e.g., text on signs and billboards in landscape photographs), or from subtitled text superimposed on images (e.g., text in television broadcasts). In addition, OCR can also refer to the general sense of recognizing text from rasterized images.

[0057] Good software: Good software is good software that is not designed to harm your computer.

[0058] Window: The basic unit of a GUI, managed by an operating system (OS) with a GUI (e.g., Microsoft Windows, MacOS, Unix / Linux-based graphical OS) to interact with the user.

[0059] Control or interactive GUI element: For Windows, a control is a child window that an application uses in conjunction with other windows to enable user interaction. For other OS or generally speaking, an interactive GUI element is a GUI element provided by a software application to enable the user to interact with the software application. Therefore, the terms "control" and "interactive GUI element" are used interchangeably in this article.

[0060] As mentioned above, malicious software or malware is a serious threat to modern computers or any other computing device. If a program officially provided by a company secretly undermines the interests of the computer user, it can be considered malware. For example, Sony sold the Sony Rootkit, which contained a Trojan horse embedded into a CD, which silently installed and hid on the purchaser's computer with the purpose of preventing illegal copying. It also reported on the user's listening habits and inadvertently created vulnerabilities that were then exploited by unrelated malware.

[0061] One strategy for preventing malware is to prevent it from gaining access to the target computer. Therefore, in addition to checking for the presence of malware and malicious activity and recovering from attacks, antivirus software, firewalls, and other strategies are used to help prevent malware from invading.

[0062] The study of malware behavior can be used to develop anti-malware solutions to prevent malware attacks. For example, one of the anti-malware solutions that has been developed is sandbox technology, which is often used to execute untrusted programs in a strictly controlled environment without harming the host device.

[0063] In computer security, a sandbox is a security mechanism used to physically or logically / virtually separate running programs, usually to mitigate the spread of system failures or software vulnerabilities. It is often used to execute untested or untrusted programs or code without harming the host or operating system, which may come from unverified or untrusted third parties, vendors, users, or websites. Sandboxes typically provide a strictly controlled set of resources for guest programs to run, such as temporary space on disk and memory. Typically, network access, the ability to inspect the host system, or read data from input devices is prohibited or severely restricted.

[0064] A sandbox can be considered a specific example of virtualization in the sense that it provides a highly controlled environment. Sandbox testing is frequently used to test unverified programs that may contain viruses or other malicious code without allowing the software to harm the host device.

[0065] Figure 1A and Figure 1B is a diagram showing an exemplary resource graph with and without a sandbox. Figure 1A and Figure 1B As shown, computer resources (such as storage capacity, network bandwidth, or processor time slices) displayed in the form of a resource grid can be allocated to multiple software applications or processes executed on the computer. Without a sandbox, the installed malware can read / write data generated / managed by other applications, invade the memory space of other software applications, or excessively occupy precious processing power and / or network bandwidth, etc., which may cause losses to computer users in a certain sense.

[0066] For example, Figure 1A As shown in , installed and executed malware (e.g., ransomware) can encrypt data managed by other applications and stored in the computer's storage device and then destroy the original data. This leaves the data owner with no choice but to pay a ransom for their data. Figure 1B As shown, when malware is installed and executed in a sandbox on a computer, the sandbox can contain the malware within a specified resource range, such as a specified storage area 100 allocated to the sandbox. In this way, resources of other applications can be protected and ensured.

[0067] Therefore, for large-scale malware behavior research, automated techniques without user interaction are usually developed in sandboxes to improve work efficiency. However, malware is constantly evolving and involves complex graphical user interfaces (GUIs) before exhibiting malicious behaviors. In order to drive malware, automated techniques must locate the correct triggers or interactive GUI elements in the malware's GUI to drive the malware to display its behaviors.

[0068] Figure 22 is a diagram showing an exemplary Windows dialog box 200 generated by a software application installed and executed in a sandbox. Figure 2 As shown, the dialog box 200 may include one or more GUI elements, such as a button 210, a toggle button 220, links 230 and 240, a close button 250, and a dialog box title 260. For such a dialog box 200, the automation technology should identify the correct trigger or interactive GUI element to drive the software application to display its behavior. Figure 2 In the particular embodiment shown, the correct interactive GUI element might be a “Run” or “Don't Run” button 210 .

[0069] Current automation solutions rely on standard APIs provided by the OS or sandbox (e.g., Windows enumeration API) to enumerate dialog boxes and find controls that require user interaction (e.g., text input, button click, or dial rotation). However, this solution may have the following drawbacks:

[0070] (1) The standard API can find controls in the dialog box, but cannot determine which control will correctly drive the malware to the next step; and / or

[0071] (2) Malware often uses obfuscation techniques to bypass automated mechanisms in sandboxes, such as layered Windows.

[0072] Therefore, a solution for locating the correct trigger or interactive GUI element in the GUI is needed. In some embodiments of the present disclosure, a heuristic solution is designed to make the sandbox click the dialog box and drive the malware to the next step, so that more behaviors can be observed. Figure 3 Combined with Figure 2 Give a detailed description of this solution.

[0073] Figure 3 is a flowchart illustrating an exemplary method of automatically determining interactive GUI elements to be interacted with in a graphical user interface (GUI) according to an embodiment of the present disclosure.

[0074] The method may begin at step 310, in which a software application may be executed or run in a sandbox. The sandbox may monitor the execution of the software application and detect any GUI events associated with the software application. In certain embodiments, the sandbox may monitor any system function calls associated with the GUI (e.g., Windows' MessageBoxIndirectA function, etc.) to detect window creation events. If a GUI event is detected at step 320, the method proceeds to step 340. Otherwise, the method may proceed to step 330, at which the method ends. In some other embodiments, the method may loop back to step 320 again to repeat detection until a GUI event is detected. Repeated detection may be performed at intervals of a certain period of time from the previous detection or may be performed immediately.

[0075] At step 340, if a GUI event is detected, the method may trigger Figure 3 The interactive GUI element detection process 350 is shown in the dashed box. Generally, the interactive GUI element detection process can use multiple interactive GUI element detection algorithms to detect candidate interactive GUI elements for selection, and select one of them as the most likely interactive GUI element to be activated.

[0076] For example, Figure 3 As shown, the interactive GUI element detection process 350 can determine candidate interactive GUI elements by using three different algorithms: an API-based algorithm 353a, a density-based algorithm 353b, and an entropy-based algorithm 353c. However, the present disclosure is not limited thereto. For example, in some other embodiments, more or fewer algorithms may be used, different algorithms may be used, and / or any of the three algorithms may be excluded from the process. In other words, the interactive GUI element detection process 350 is shown for purposes of illustration only and not limitation. Figure 3 The following will refer to the three algorithms described in Figures 4 to 9 The three algorithms are explained in detail. The outputs of these three algorithms 353a-353c can be fed into a voting algorithm (at step 355) to determine the interactive GUI element with the highest probability to drive the malware to the next step.

[0077] First, refer to Figure 4 and Figure 5 The API-based algorithm 353a is described in detail. Figure 4 is a flowchart illustrating an exemplary method of determining interactive GUI elements by using an API-based algorithm according to an embodiment of the present disclosure. Figure 5 is a diagram illustrating an exemplary GUI 500 to be processed by an API-based algorithm according to an embodiment of the present disclosure.

[0078] like Figure 5As shown, GUI 500 may include one or more windows 510 and 520 and some desktop icons 530a-530c. However, the present disclosure is not limited thereto, and GUI 500 may include any number and / or any type of GUI elements. Figure 5 In the illustrated embodiment, window 510 "Software Installation" is a window generated by the software in question, and window 520 "Notification" is generated by another software application. API-based algorithm 353a is used to determine candidate interactive GUI elements in GUI 500 by using an API provided by the OS or sandbox.

[0079] Figure 4 The method shown may start from step 410, in which the top window belonging to the software is determined. In some embodiments, the software may generate multiple windows when executed. For example, if the software is an instant messaging tool, the instant messaging tool may generate multiple chat windows for multiple friends or buddies respectively. In order to drive the instant messaging tool to the next step, for example, the focus window is determined by calling the API function "GetFocus". Back Figure 5 To determine the top window belonging to the software in question, an API function call (e.g., "GetFocus") may be used to determine the window with focus (e.g., Figure 5 In some other embodiments, for the determined focus window, it is further determined whether the window belongs to the software, for example, by checking its parent window ID or process ID. If the window belongs to the software, the method proceeds to step 420. However, if the window does not belong to the software, the method or algorithm 353a may simply return an empty result or indicator to, for example, Figure 3 Voting algorithm 355 is shown.

[0080] In addition, in some other embodiments where the window does not belong to the software, algorithm 353a can select one of the windows generated by the software as the top window randomly or according to predetermined rules (e.g., the last generated window) by using an API function call (e.g., "EnumWindows").

[0081] At step 420, the method may enumerate the controls (or interactive GUI elements) in the top window, for example, by using an API function call (e.g., "EnumChildWindows"). If it is determined at step 430 that no such control exists, the method proceeds to step 440, where the method ends. Otherwise, the method proceeds to step 450, where the text associated with the control is extracted from the control, for example, by one or more API function calls (e.g., "GetWindowTextA", "ComboBox_GetText", and / or "Button_GetText", depending on the particular type of control).

[0082] Next, at step 460, the method may collect the position and text of the control and check if there are any more controls. If there is another control in the top window, the method loops back to step 430 and repeats steps 430 to 460. If it is determined in step 460 that there are no more controls, the method or algorithm 353a may return a list of tuples of six elements, such as (algorithm_type, parent_window, position, text, density, entropy). In this embodiment, all three algorithms use the format of the tuple, so there are fields of other algorithms such as "density" and "entropy" that are not used by algorithm 353a. In this case, these two fields may be set to default values, such as 0. In addition, in some other embodiments, algorithm 353a may return its output in another format, such as (position, text) collected in step 460, without other fields. Therefore, the format of the tuple used in this embodiment is only an example and does not limit the scope of the present disclosure.

[0083] A specific example of a tuple may be ("standard_api_method", 0x234, (100, 122, 50), "next", 0, 0). In this example, the value of the algorithm_type field is "standard_api_method", indicating that the tuple is returned by an API-based method. In addition, the parent_window field has a value of "0x234", indicating that the control corresponding to the tuple belongs to a parent window with Windows ID "0x234". In this way, subsequent steps of the method can distinguish different windows generated by the software from each other. In addition, the position field has a value of "(100, 122, 50)", which indicates the position of the control on the screen or in the parent window. In other words, the position field may have absolute or relative coordinates of the control. In addition, the value of the text field is "next", which indicates that the control has the text "next" associated with it or marked on it. In addition, the two fields of "density" and "entropy" will be explained later when describing other algorithms. In addition, the present disclosure is not limited to the above specific values.

[0084] In addition, in some embodiments, the method can also select some of the detected controls instead of unconditionally selecting all of them. For example, as will be described in detail in algorithm 353b, a likelihood value can be assigned to each word or phrase to indicate the likelihood of the word or phrase being associated with an interactive GUI element that can drive the software to the next step. For example, the word "next" can be assigned a likelihood value of 1, and the word "cancel" can be assigned a likelihood value of "0.6". Based on the likelihood values, the method can select those detected controls that have likelihood values ​​greater than a predetermined threshold (e.g., 0.8).

[0085] use Figure 4 The method shown or Figure 3 The algorithm 353a shown can collect some text information of the software and its location information, which can guide the user to interact with the GUI of the software. However, consider the case where the text information cannot be captured through API function calls. For example, instead of displaying a button with text to the user, a hyperlink with a picture is displayed to the user, and the text information is displayed on the picture (the text information cannot be directly obtained through API function calls), so that the user is guided by the text information on the picture to interact with the hyperlink. In this case, the API-based method is not very useful.

[0086] Therefore, some other algorithms that can extract text information in this case are described below (e.g., algorithm 353b "density-based algorithm" and algorithm 353c "entropy-based algorithm"). None of these algorithms rely on APIs provided by the operating system or sandbox.

[0087] Reference again Figure 3 , before executing algorithm 353b or 353c, the method may convert the window into an image at step 351. For example, given a window handle (e.g., Figure 4 The method can determine the window coordinates and size information of the window and obtain the focus of the window in a similar manner to step 410. Then, the method can capture or obtain the rectangular area of ​​the window as an image, for example, by printing the rectangular area as an image file. In addition, the rectangular area can also be rasterized in other ways, and the present disclosure is not limited thereto. The image file will be used by algorithms 353b and 353c.

[0088] Next, we will refer to Figure 6 and Figure 7 The density-based algorithm 353b is described in detail. Figure 6 is a flow chart illustrating an exemplary method of determining interactive GUI elements by using a density-based algorithm according to an embodiment of the present disclosure. Figure 7 is a diagram illustrating an exemplary GUI 700 to be processed by a density-based algorithm according to an embodiment of the present disclosure.

[0089] like Figure 7 As shown, GUI 700 may include one or more windows 710 and 720 and some desktop icons 730a-730c. However, the present disclosure is not limited thereto, and GUI 700 may include any number and / or any type of GUI elements. Figure 7 In the illustrated embodiment, window 710 "Software Installation" is a window generated by the software of interest, and window 720 "Notification" is generated by another software application. Density-based algorithm 353b may be based on the Figure 3 The density of the text block of the image converted in step 351 is used to determine the candidate interactive GUI elements in GUI 700.

[0090] Figure 6 The method shown may start at step 610, where a window visible on the screen (e.g., a full-screen window at the top or two side-by-side windows at the top, etc.) may be found or determined (e.g., by a pattern recognition method). In addition, as previously described, another method (e.g., similar to Figure 4 Step 410 shown in FIG. 4A ) determines the window. Figure 7, window 710 may be determined as the top window, and window 720 may not be selected because window 720 is partially covered by window 710, which indicates that window 720 is not at the top or does not have focus.

[0091] Next, at step 620 , the rectangular area of ​​window 710 may be calculated or otherwise determined.

[0092] At step 630, the method may detect text blocks in the image of the rectangular area of ​​the window 710 by using an OCR method and extract text information from the text blocks. Then, at step 640, for each text block, density is calculated based on the text extracted therefrom.

[0093] In some embodiments, the density is the ratio of the "target" word to all words in the text block. For example, the text in the block may be "Notepad release setup program" with 4 words, and only "setup" is the target word, which the user is more likely to click. Therefore, the density can be calculated as 1 / 4=0.25. For another example, if Figure 7 As shown, the density of text block 711a may be 1 because there is only one word "next" in text block 711a; the density of text block 710a may be 0 because there is no target word but two non-target words in text block 710a.

[0094] In some embodiments, a non-exhaustive set of "target" words may include: "next", "install", "update", "upgrade", "upgrade now", "next", "get stated", "start", and "run", all of which may have a likelihood value of 1. In addition, another non-exhaustive set of words may include: "later" (with a likelihood value of 0.6), "ignore" (with a likelihood value of 0.6), and "ok" (with a likelihood value of 0.4). Please note that the likelihood values ​​corresponding to the target words will be used later in step 355, and therefore, a detailed description thereof will be given there.

[0095] For each text block having a density greater than a predetermined threshold (e.g., 0.6), a six-element tuple identical to the above tuple may be returned as output of algorithm 353b to Figure 3 method.

[0096] A specific example of a tuple may be ("image_text_recognition", 0x234, (100, 100, 30), "next", 0, 1). It is similar to the tuple returned by algorithm 353a, so only the differences between them are described in detail. In this example, the value of the algorithm_type field is "image_text_recognition", indicating that the tuple is returned by a density-based method. In addition, the text field has a value of "next", indicating that the text block has the text "next" associated with it or marked on it. In addition, the value of the density field is 1, indicating that the text block is likely to correspond to a clickable button or interactive GUI element. The intuition behind the algorithm is that blocks with higher density may be more easily clicked by users.

[0097] Next, we will refer to Figure 8 and Fig. 9 The entropy-based algorithm 353c is described in detail. Figure 8 is a flow chart illustrating an exemplary method of determining interactive GUI elements by using an entropy-based algorithm (eg, algorithm 353b) according to an embodiment of the present disclosure. Fig. 9 is a diagram illustrating another exemplary GUI 900 to be processed by an entropy-based algorithm according to an embodiment of the present disclosure.

[0098] like Fig. 9 As shown, GUI 900 may include one or more windows 910 and 920 and some desktop icons 930a-930c. However, the present disclosure is not limited thereto, and GUI 900 may include any number and / or any type of GUI elements. Fig. 9 In the illustrated embodiment, window 910 "Software Installation" is a window generated by the software of interest, and window 920 "Notification" is generated by another software application. API-based algorithm 353b can be based on Figure 3 The entropy of each portion of the image converted in step 351 is used to determine candidate interactive GUI elements in GUI 900.

[0099] Figure 8 The method shown may start at step 810, where a window visible on the screen (e.g., a full-screen window at the top or two side-by-side windows at the top, etc.) may be found or determined (e.g., by a pattern recognition method). In addition, as previously described, another method (e.g., similar to Figure 4 Step 410 shown in FIG. 4A ) determines the window. Fig. 9, window 910 may be determined as the top window, and window 920 may not be selected because window 920 is partially covered by window 910, which indicates that window 920 is not at the top or does not have focus.

[0100] Next, at step 820, the rectangular area of ​​window 910 may be calculated or otherwise determined. Based on the calculation or determination, the image of the rectangular area may be segmented into a plurality of blocks at step 830. For example, Fig. 9 As shown, the image of window 910 is divided into M×N blocks, where M and N are positive integers and can be predetermined or dynamically determined according to the size of the image. In a specific embodiment, M and N are both 10.

[0101] Next, at step 840, entropy may be calculated for each block. In some embodiments, entropy may be calculated based on the intensity or edge orientation of the pixels in the block (this is known in the art and omitted for simplicity), and such entropy may generally indicate how much information related to intensity or edge orientation is contained in the block. In other words, blocks with higher entropy are more likely to contain interactive GUI elements than blocks with lower entropy. Fig. 9 , window 910 is divided into a plurality of blocks, and buttons 911 and 913 fall into some blocks, such as 911a, 911b, 913a, and 913b, etc. Since block 911a contains some text information of button 911, its entropy (which can be calculated based on the intensity of the pixels in block 911a) can be higher than other blocks (such as block 911b). In addition, in some other embodiments, since block 911b contains the edge of window 910, its entropy (which can be calculated based on the edge orientation of the pixels in block 911b) can be higher than other blocks (such as block 911a). That is, depending on how to calculate entropy, blocks may have different entropies. In some embodiments, entropy may fall within the range of [0,1]. In some other embodiments, entropy may be greater than 1, and if necessary, it may be normalized to the range of [0,1].

[0102] For each block having an entropy greater than a predetermined threshold (e.g., 0.8), a six-element tuple identical to the above tuple may be returned as output of algorithm 353c to Figure 3 method.

[0103] A specific example of a tuple may be ("entropy_method", 0x234, (100, 122, 40), "next", 0.3, 0). It is similar to the tuples returned by algorithm 353a and algorithm 353b, so only the differences between them are described in detail. In this example, the value of the algorithm_type field is "entropy_method", indicating that the tuple is returned by an entropy-based method. In addition, the value of the text field is "next", which indicates that the control has the text "next" associated with it or marked on it. In addition, the entropy field has a value of "0.3", which indicates that the block may have some but not too much information related to intensity or edge orientation.

[0104] In addition, algorithm 353c can return to Figure 3 Sort all the blocks of the method and provide them to in descending order of entropy Figure 3 method.

[0105] Return to reference Figure 3 After all algorithms provide their outputs, the method may proceed to step 355. At step 355, a voting algorithm may be used to determine which of the outputs from all algorithms is the interactive GUI element to be interacted with by the sandbox.

[0106] An example of a voting algorithm is given below, which is for illustration only.

[0107] 1. Candidates = Empty list

[0108] 2. For each (algorithm type, position, text, entropy, density) tuple in the list, do the following

[0109] 3. If Algorithm Type is "Standard API Method", and Likelihood(Text) > LikelihoodThreshold:

[0110] 4. Add (position, likelihood (text) + weight value of algorithm 1, parent window, text) to the candidates

[0111] 5. If the algorithm type is "Image Text Recognition", and Density (Text) > Density Threshold and Likelihood (Text) > Likelihood Threshold:

[0112] 6. Add (position, likelihood (text) + weight value of algorithm 2, parent window, text) to candidates

[0113] 7. If the algorithm type is "Entropy Method", and Entropy > Entropy Threshold:

[0114] 8. Add (position, entropy + weight value of algorithm 3, parent window, text) to candidates

[0115] 9. Sort candidates according to likelihood

[0116] (1. Candidates = empty_list

[0117] 2. For each (algorithm_type, position, text, entropy, density) tuple in list do

[0118] 3. If algorithm_type is “standard_api_method” and likelihood(text) > likelihood_threshold:

[0119] 4. Add (position, likelihood(text) + weight_value_of_algorithm1, parent_window, text) to candidates

[0120] 5. if algorithm_type is “image_text_recognition” and density(text) > density_threshold and likelihood(text) > likelihood_threshold:

[0121] 6. Add (position, likelihood(text) + weight_value_of_algorithm2, parent_window, text) to candidates

[0122] 7. if algorithm_type is “entropy_method” and entropy > entropy_threshold:

[0123] 8. Add (position, entropy + weight_value_of_algorithm3, parent_window, text) to candidates

[0124] 9. sort candidates by the likelihood)

[0125] In the above algorithm, the "likelihood threshold" may be 0.8, the "density threshold" may be 0.6, and the "entropy threshold" may be 0.8. In addition, in the above algorithm, the "weight value of algorithm 1", the "weight value of algorithm 2", and the "weight value of algorithm 3" may be 0.03, 0.02, and 0.01, respectively. However, the present disclosure is not limited thereto, and any one, some, or all of these values ​​may be set empirically.

[0126] According to the above voting algorithm, the method can selectively decide whether to consider the output of algorithm 353a / 353b / 353c for voting algorithm selection. For example, for API-based algorithm 353a, only controls with likelihood values ​​greater than 0.8 can be considered for voting algorithm selection. For another example, for density-based algorithm 353b, only text blocks with a density greater than 0.6 and a likelihood value greater than 0.8 can be considered for voting algorithm selection. For another example, for entropy-based algorithm 353c, only blocks with entropy values ​​greater than 0.8 can be considered for voting algorithm selection. However, the present disclosure is not limited to this.

[0127] In addition, since different algorithms may have different outputs within different value ranges, and these outputs can be adjusted before they can be compared to each other in a meaningful way. Therefore, weights can be assigned to the outputs of each algorithm to balance the outputs from different algorithms. However, the present disclosure is not limited to this. In some other embodiments, weighting can be performed in another way different from "adding weights", for example, by multiplying a factor (weight), by a linear function, etc. In addition, candidate interactive GUI elements can be sorted according to other characteristics other than likelihood values, and the present disclosure is not limited to this.

[0128] Once the voting algorithm has determined the most promising interactive GUI element at step 355, the method may optionally check at step 357 whether the determined interactive GUI element has been previously recorded in the database. Fig.10 Describe this step in detail.

[0129] If this element has not been previously recorded in the database, or if step 357 is omitted (because it is optional), the sandbox can interact with the element to drive the software to the next step at step 320, and wait to see or observe whether there is another GUI event. Thereafter, the method can repeat steps 320 to 360. In this way, a fully automatic interactive GUI element detection mechanism is possible, and thus a malware behavior automatic analysis system can be established.

[0130] Furthermore, in the pseudo-code above, the candidate selection function is used to aggregate the outputs of the previously mentioned algorithms, filter out unqualified positions, and retain the top positions that have a high probability of driving the software to run.

[0131] Furthermore, the likelihood corresponding to the text can be established based on empirical studies of both good software and malware. For example, a button with the text "next" is more likely to be clicked by a user than "close". Furthermore, the text can be a word in any language.

[0132] Fig.10 The present invention is shown in Figure 3 , which is a flowchart of an exemplary embodiment of some steps of the method shown in . As described above, once the voting algorithm has determined the most promising interactive GUI element, the method can check at step 257 whether the element has been previously recorded. If the answer is yes, that is, the element has a match in the database, then the method can return to the voting algorithm 255 to check the next element in the voting list, and repeat the process until an element is found that is not recorded in the database. If all possible elements have been recorded in the database, the method can proceed directly to end 330 ( Figure 3 to end the whole process. Fig.10 As shown, one particular method for checking whether the element has been previously recorded is to compare it with an entry in a database that stores historical information of previously interacted interactive GUI elements. A simple algorithm is provided below, which is for illustration only.

[0133]

[0134]

[0135] The drive_window function accepts the input of the top position, gets the window that position is in, and simulates a mouse click on that position. The sandbox continuously monitors the malware behavior if the window state changes, such as the window is active / enabled / visible, etc.

[0136] Furthermore, based on the above pseudo code, it is obvious that the location information is used as an index or primary key of the data in the database. However, the present disclosure is not limited thereto.

[0137] In addition, Fig.10 As shown, if the most promising interactive GUI element has not been previously recorded in the database, it can be added to the database in step 358 for subsequent processing. In some embodiments, step 358 can be combined with Figure 3 The step 360 is performed in parallel with, before, or after the step 360, and the present disclosure is not limited thereto.

[0138] Typically, a database can be used to track which window has been previously visited and / or its state has changed. It helps to some extent to eliminate the problem of repeated navigation to the same window. In some embodiments, each record / entry in the database can uniquely identify the current window. For example, a record can contain the following information:

[0139] Parent window ID; for example, 0x234

[0140] The current window title; for example, "Notepad Settings"

[0141] Current window coordinates (x, y); for example (125, 195)

[0142] Current window size (height x width); for example (500 x 500)

[0143] The clicked position in the window; for example, two positions in the list [(100, 100), (120, 130)]

[0144] The solutions proposed above are designed to deal with complex malware that use GUI interfaces to evade sandbox detection. If the sandbox cannot find the right place to drive the malware to the next step, the dynamic behavior of the malware cannot be observed. Advanced anti-malware detection methods (such as behavior-based methods) require more behavioral data from malware and good software. Therefore, the methods proposed above can provide rich behavioral data when malware / good software involves GUI.

[0145] In summary, the above-proposed solutions according to some embodiments of the present application utilize standard API methods and heuristic methods (e.g., image recognition (entropy-based) and OCR-based techniques) to identify locations in the GUI interface that will cause the malware to run to the next step.

[0146] Fig.11 is a block diagram illustrating an exemplary arrangement that may be used in an electronic device according to an embodiment of the present disclosure.

[0147] like Fig.11 As shown, the electronic device or computing device 1110 can be a server computer, a cluster, a cloud computer, a general-purpose computer, a mobile device, a tablet computer or a dedicated computer, which can detect one or more candidate interactive GUI elements in the GUI based on multiple algorithms; determine a likelihood indicator for each of the one or more candidate interactive GUI elements, the likelihood indicator indicating the likelihood that the candidate interactive GUI element associated with the likelihood indicator is the interactive GUI element to be interacted with; and determine the interactive GUI element to be interacted with from the one or more candidate interactive GUI elements based on the likelihood indicator.

[0148] In some embodiments, computing device 1110 can communicate with other computing devices or services to obtain necessary data from those computing devices to perform the above methods and provide results to those computing devices. In some embodiments, communication can be performed via a network, which can be a wired or wireless network and can have various forms, such as a public network and / or a private network.

[0149] like Fig.11 As shown, computing device 1110 may include, but is not limited to, processor 1112, memory 1114, and optional storage device 1116. In some embodiments, computing device 1110 may include other hardware components and / or software components (not shown) to perform its corresponding tasks. Examples of these hardware and / or software components may include, but are not limited to, other required memories, interfaces, buses, input / output (I / O) modules or devices, network interfaces, and / or peripheral devices.

[0150] The processor 1112 may be a central processing unit (CPU) configured to control the operation of the computing device 1110. The processor 1112 may execute an operating system (OS) or other applications of the computing device 1110. In some embodiments, the computing device 1110 may have multiple CPUs as processors, such as two CPUs, four CPUs, eight CPUs, or any suitable number of CPUs.

[0151] The memory 1114 may be a volatile memory, such as a random access memory (RAM), for storing data and / or information during operation of the computing device 1110. In some embodiments, the memory 1114 may be a volatile memory array. In some embodiments, the computing device 1110 may run on more than one memory 1114. The optional storage device 1116 may be a non-volatile data storage medium for storing an OS (not shown) and other applications of the computing device 1110. Examples of the storage device 1116 may include non-volatile memory, such as flash memory, a memory card, a USB drive, a hard drive, a floppy disk, an optical drive, a solid-state drive (SSD), or any other type of data storage device. In some embodiments, the storage device 1116 may be local storage, remote storage, or cloud storage. In some embodiments, the computing device 1110 may have multiple storage devices 1116, which may be the same storage device or different types of storage devices, and the applications of the computing device 1110 may be stored in one or more storage devices 1116 of the computing device 1110. In some embodiments, computing device 1110 is a cloud computer, and processor 1112, memory 1114, and storage device 1116 are shared resources provided on demand over the Internet.

[0152] like Fig.11As shown, storage device 1116 may include interactive GUI element detection application 1118, as well as algorithm-specific likelihood data 1190, algorithm-specific thresholds 1192, algorithm-specific weights (not shown), likelihood indicator data 1194, and interactive GUI element history database 1196. Interactive GUI element detection application 1118 may be configured to detect interactive GUI elements to be interacted with by a sandbox or computer.

[0153] Interactive GUI element detection application 1118 may include detection module 1120, likelihood indicator determination module 1140 and interactive GUI element determination module 1160, etc. In some embodiments, interactive GUI element detection application 1118 may include other applications or modules necessary for the operation of interactive GUI element detection application 1118. It should be noted that each module is implemented by computer executable code or instructions, or data tables or databases, which together form an application. In some embodiments, each module may further include submodules. Alternatively, some modules may be combined into a stack. In other embodiments, some modules may be implemented as circuits rather than executable codes. In some embodiments, some or all modules of interactive GUI element detection application 1118 may be located at a remote computing device or distributed in the cloud.

[0154] In some embodiments, the detection module 1120 can be configured to detect one or more candidate interactive GUI elements in the GUI based on a plurality of algorithms. Figure 3 As shown in steps 353a-353c, multiple algorithms are used to determine candidate interactive GUI elements.

[0155] In some embodiments, the likelihood indicator determination module 1140 may be configured to determine a likelihood indicator for each of one or more candidate interactive GUI elements, wherein the likelihood indicator may indicate the likelihood that the candidate interactive GUI element associated with the likelihood indicator is the interactive GUI element to be interacted with. For example, as shown in step 355 and its description, a likelihood value or indicator is calculated or assigned for each candidate interactive GUI element. More specifically, for an API-based algorithm, the likelihood indicator of a control or candidate interactive GUI element may be determined based on the likelihood of the text information of the control or candidate interactive GUI element and an algorithm-specific weight. For an entropy-based algorithm, the likelihood indicator of a block or candidate interactive GUI element may be determined based on the entropy of the block and an algorithm-specific weight. For a density-based algorithm, the likelihood indicator of a text block or candidate interactive GUI element may be determined based on the likelihood of the text block and an algorithm-specific weight.

[0156] In some embodiments, the interactive GUI element determination module 1160 may be configured to determine the interactive GUI element to be interacted from one or more candidate interactive GUI elements based on the likelihood indicator. For example, as shown in step 355 and its description, the candidate interactive GUI element with the highest likelihood may be determined as the interactive GUI element to be interacted.

[0157] As mentioned above, these modules can be implemented as instructions or codes to be executed on the processor 1112, so the above modules can be described in another way. For example, the memory 1114 can store instructions that, when executed by the processor 1112, cause the processor 1112 to: detect one or more candidate interactive GUI elements in the GUI based on multiple algorithms; and determine a likelihood indicator for each of the one or more candidate interactive GUI elements, the likelihood indicator indicating the likelihood that the candidate interactive GUI element associated with the likelihood indicator is the interactive GUI element to be interacted with; and determine the interactive GUI element to be interacted with from the one or more candidate interactive GUI elements based on the likelihood indicator.

[0158] In some embodiments, the plurality of algorithms may include at least one of: an algorithm based on an application programming interface (API), an algorithm based on optical character recognition (OCR), and an algorithm based on entropy. In some embodiments, the memory 1114 may store instructions that, when executed by the processor 1112, further cause the processor 1112 to extract text from each of one or more interactive GUI elements in the GUI by using the API for the GUI; determine a first algorithm-specific likelihood for each of the one or more interactive GUI elements based on the extracted text; and for each interactive GUI element whose calculated first algorithm-specific likelihood is greater than a first algorithm-specific threshold, determine the interactive GUI element as a candidate interactive GUI element.

[0159] In some embodiments, the memory 1114 may store instructions that, when executed by the processor 1112, further cause the processor 1112 to compare the extracted text with one or more texts in a predefined set of texts; and in response to the comparison result indicating that the extracted text matches a text, determine the likelihood corresponding to the text in the predefined set of texts as the first algorithm-specific likelihood of the corresponding interactive GUI element. In some embodiments, the memory 1114 may store instructions that, when executed by the processor 1112, further cause the processor 1112 to determine the likelihood indicator by weighting the first algorithm-specific likelihood with a first algorithm-specific weight.

[0160] In some embodiments, the memory 1114 may store instructions that, when executed by the processor 1112, further cause the processor 1112 to obtain a GUI image corresponding to the GUI; segment the GUI image into a plurality of blocks; determine the entropy of each of the plurality of blocks by using an entropy-based algorithm; for each block whose entropy is greater than a threshold value specific to the second algorithm, determine the interactive GUI element associated with the block as a candidate interactive GUI element. In some embodiments, the memory 1114 may store instructions that, when executed by the processor 1112, further cause the processor 1112 to determine the entropy value of each of the plurality of blocks based on the intensity and / or edge orientation of the pixels in the corresponding block in the GUI image. In some embodiments, the memory 1114 may store instructions that, when executed by the processor 1112, further cause the processor 1112 to determine the likelihood indicator by weighting the entropy with a weight specific to the second algorithm.

[0161] In some embodiments, the memory 1114 may store instructions that, when executed by the processor 1112, further cause the processor 1112 to detect one or more text blocks in the GUI by using an OCR-based algorithm; for each of the one or more text blocks, calculate a density and / or a third algorithm-specific likelihood based on the text in the corresponding text block; for each text block having a density greater than a third algorithm-specific threshold and / or a third algorithm-specific likelihood greater than a first algorithm-specific threshold, determine an interactive GUI element associated with the block as a candidate interactive GUI element. In some embodiments, the memory 1114 may store instructions that, when executed by the processor 1112, further cause the processor 1112 to determine the number of target words and the number of all words in the text in the corresponding text block; and then calculate the density by dividing the number of all words by the number of target words.

[0162] In some embodiments, the memory 1114 may store instructions that, when executed by the processor 1112, further cause the processor 1112 to determine a likelihood indicator by weighting the third algorithm-specific likelihood with a third algorithm-specific weight. In some embodiments, the memory 1114 may store instructions that, when executed by the processor 1112, further cause the processor 1112 to sort one or more candidate interactive GUI elements according to the likelihood indicator; and determine the candidate interactive GUI element with the highest likelihood indicator as the interactive GUI element to be interacted with.

[0163] In some embodiments, the memory 1114 may store instructions that, when executed by the processor 1112, further cause the processor 1112 to compare the information related to the determined interactive GUI element with the historical information previously stored in the database. In some embodiments, the memory 1114 may store instructions that, when executed by the processor 1112, further cause the processor 1112 to interact with the determined interactive GUI element by simulating a user operation corresponding to the type of the interactive GUI element in response to determining that the information related to the determined interactive GUI element to be interacted does not match the historical information previously stored in the database. In some embodiments, the memory 1114 may store instructions that, when executed by the processor 1112, further cause the processor 1112 to determine whether the state of the GUI changes after the interaction; and, in response to determining that the state of the GUI is changed after the interaction, record the information related to the determined interactive GUI element in the database as historical information.

[0164] In some embodiments, the memory 1114 may store instructions that, when executed by the processor 1112, further cause the processor 1112 to determine another interactive GUI element to be interacted with from one or more candidate interactive GUI elements based on the likelihood indicator in response to determining that the information related to the determined interactive GUI element to be interacted with matches the historical information previously stored in the database. In some embodiments, the information related to the determined interactive GUI element to be interacted with includes at least one of the following: an identifier of the GUI; a title of the determined interactive GUI element to be interacted with; a position of the determined interactive GUI element to be interacted with; a size of the determined interactive GUI element to be interacted with; and a position where the interaction is to occur. In some embodiments, an application corresponding to the GUI is executed in a sandbox.

[0165] In summary, certain embodiments of the present disclosure provide a solution for automatically detecting interactive GUI elements to be interacted with. In addition, corresponding devices and non-transitory computer-readable storage media are also provided.

[0166] The foregoing description of exemplary embodiments of the present disclosure has been presented for purposes of illustration and description only and is not intended to be exhaustive or to limit the disclosure to the precise form disclosed.Many modifications and variations are possible in light of the above teachings.

[0167] The embodiments are selected and described in order to explain the principles of the present disclosure and its practical application, so as to enable other persons skilled in the art to utilize the present disclosure and the various embodiments and make various modifications to suit the intended specific use. Alternative embodiments will become clear to those skilled in the art to which the present disclosure belongs without departing from the spirit and scope of the present invention. Therefore, the scope of the present disclosure is limited by the appended claims, rather than by the foregoing description and the exemplary embodiments described therein.

Claims

1. A computer-implemented method for automatically determining interactive GUI elements to be interacted with in a graphical user interface (GUI), the method comprising: The processor detects one or more candidate interactive GUI elements in the GUI based on a plurality of algorithms; The processor determines, for each of the one or more candidate interactive GUI elements, a likelihood indicator indicating a likelihood that the candidate interactive GUI element associated with the likelihood indicator is an interactive GUI element to be interacted with; The processor determines an interactive GUI element to be interacted with from the one or more candidate interactive GUI elements based on the likelihood indicator; The processor compares information related to the determined interactive GUI element to be interacted with historical information previously stored in a database; as well as In response to determining that information related to the determined interactive GUI element to be interacted with does not match historical information previously stored in the database, the processor interacts with the determined interactive GUI element to be interacted with by simulating a user operation corresponding to the type of the interactive GUI element to be interacted with.

2. The method according to claim 1, wherein the plurality of algorithms comprises at least one of the following: Algorithms based on application programming interface API; Entropy-based algorithms; and Algorithm based on Optical Character Recognition (OCR).

3. The method of claim 2, wherein the step of the processor detecting one or more candidate interactive GUI elements in the GUI based on a plurality of algorithms comprises: The processor extracts text from each of one or more interactive GUI elements in the GUI by using an API of the GUI; the processor determining a first algorithm-specific likelihood for each of the one or more interactive GUI elements based on the extracted text; as well as For each interactive GUI element for which the calculated first algorithm-specific likelihood is greater than the first algorithm-specific threshold, the processor determines the interactive GUI element as a candidate interactive GUI element.

4. The method of claim 3, wherein the step of the processor determining a first algorithm-specific likelihood for each of the one or more interactive GUI elements based on the extracted text comprises: The processor compares the extracted text to one or more texts in a predefined set of texts; as well as In response to the comparison result indicating that the extracted text matches a text, the processor determines a likelihood corresponding to the text in the predefined set of texts as a first algorithm-specific likelihood of the corresponding interactive GUI element.

5. The method of claim 3, wherein the step of the processor determining a likelihood indicator for each of the one or more candidate interactive GUI elements comprises: The processor determines the likelihood indicator by weighting a first algorithm-specific likelihood with a first algorithm-specific weight.

6. The method of claim 2, wherein the step of the processor detecting one or more candidate interactive GUI elements in the GUI based on a plurality of algorithms comprises: The processor detects one or more blocks of text in the GUI by using an OCR-based algorithm; The processor calculates, for each of the one or more text blocks, a density and / or a second algorithm-specific likelihood based on text in the corresponding text block; as well as For each text block having the density greater than a second algorithm-specific threshold and / or the second algorithm-specific likelihood greater than a second algorithm-specific threshold, the processor determines the interactive GUI element associated with the text block as a candidate interactive GUI element.

7. The method of claim 6, wherein the step of the processor calculating the density for each of the one or more text blocks based on the text in the corresponding text block comprises: The processor determines the number of target words and the number of all words in the text in the corresponding text block; as well as The processor calculates the density by dividing the number of target words by the number of all words.

8. The method of claim 6, wherein the step of the processor determining a likelihood indicator for each of the one or more candidate interactive GUI elements comprises: The processor determines the likelihood indicator by weighting the second algorithm-specific likelihood with a second algorithm-specific weight.

9. The method of claim 2, wherein the step of the processor detecting one or more candidate interactive GUI elements in the GUI based on a plurality of algorithms comprises: The processor obtains a GUI image corresponding to the GUI; The processor divides the GUI image into a plurality of blocks; The processor determines entropy for each block of the plurality of blocks by using an entropy-based algorithm; as well as For each block having an entropy greater than a third algorithm-specific threshold, the processor determines an interactive GUI element associated with the block as a candidate interactive GUI element.

10. The method of claim 9, wherein the step of the processor determining entropy for each of the plurality of blocks by using an entropy-based algorithm comprises: The processor determines an entropy value for each of the plurality of blocks based on intensities and / or edge orientations of pixels in corresponding blocks in the GUI image.

11. The method of claim 9, wherein the step of the processor determining a likelihood indicator for each of the one or more candidate interactive GUI elements comprises: The processor determines the likelihood indicator by weighting the entropy with a third algorithm-specific weight.

12. The method according to claim 1, wherein the step of the processor determining the interactive GUI element to be interacted with from the one or more candidate interactive GUI elements based on the likelihood indicator comprises: The processor ranks the one or more candidate interactive GUI elements according to likelihood indicators; as well as The processor determines the candidate interactive GUI element having the highest likelihood indicator as the interactive GUI element to be interacted with.

13. The method according to claim 1, further comprising: The processor determines whether the state of the GUI changes after the interaction; as well as In response to determining that the state of the GUI is changed after the interaction, the processor records information related to the determined interactive GUI element in a database as historical information.

14. The method according to claim 1, further comprising: In response to determining that the information related to the determined interactive GUI element to be interacted matches historical information previously stored in the database, the processor determines another interactive GUI element to be interacted from the one or more candidate interactive GUI elements based on the likelihood indicator.

15. The method according to claim 1, wherein the information related to the determined interactive GUI element to be interacted with comprises at least one of the following: an identifier of the GUI; The title of the determined interactive GUI element to be interacted with; The determined position of the interactive GUI element to be interacted with; The determined size of the interactive GUI element to be interacted with; as well as The location where the interaction is to occur.

16. The method of claim 1, wherein an application corresponding to the GUI is executed in a sandbox.

17. A device for automatically determining interactive GUI elements to be interacted with in a graphical user interface GUI, comprising: processor; a memory storing instructions that, when executed by the processor, cause the processor to: detecting one or more candidate interactive GUI elements in the GUI based on a plurality of algorithms; determining, for each of the one or more candidate interactive GUI elements, a likelihood indicator indicating a likelihood that the candidate interactive GUI element associated with the likelihood indicator is an interactive GUI element to be interacted with; determining an interactive GUI element to be interacted with from the one or more candidate interactive GUI elements based on the likelihood indicator; comparing information related to the determined interactive GUI element to be interacted with with historical information previously stored in a database; as well as In response to determining that information related to the determined interactive GUI element to be interacted with does not match historical information previously stored in the database, interact with the determined interactive GUI element to be interacted with by simulating user operations corresponding to the type of the interactive GUI element to be interacted with.

18. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1.

Images