Graphical interface operation method, apparatus, device, and medium

Abstract

This application provides a graphical user interface (GUI) operation method, apparatus, device, and medium. The method includes: performing intent recognition and parsing on natural language commands to obtain a structured intent sequence; performing parsing and semantic mapping operations on the candidate GUI operation sequence to obtain a structured GUI operation sequence; constructing a matching score matrix between the structured GUI operation sequence and the structured intent sequence based on a local sequence alignment algorithm; determining the element with the largest value in the matching score matrix as the matching degree; and executing the candidate GUI operation sequence if the matching degree is greater than a threshold. This solves the semantic difference between the GUI operation sequence and the natural language commands, and robustly handles the mismatch in granularity and length between the two sequences, thereby improving the accuracy of the calculated matching degree and thus increasing the consistency between the GUI operation result and the user's expectations, enhancing the user experience.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, specifically to a graphical interface operation method, apparatus, device, and medium. Background Technology

[0002] With the development of artificial intelligence technology, Natural Language Processing (NLP) and Human-Computer Interaction (HCI) have become important components of intelligent software systems. Traditional software operation mainly relies on explicit input methods such as mouse and keyboard, while natural language interaction allows users to control the graphical user interface (GUI) in a conversational manner, thereby significantly reducing the barrier to entry and improving interaction efficiency.

[0003] In recent years, deep learning-based natural language understanding models (such as BERT and GPT, large-scale language models) have achieved remarkable results in tasks such as semantic parsing, intent recognition, and command generation. This has made "language-driven interface operations" a research hotspot in fields such as intelligent assistants, office automation, visual programming, and low-code development. For example, users can input natural language commands such as "open the settings page and enable dark mode" to allow the system to automatically execute corresponding multi-step interface operations. However, the conversion process from language to operation essentially involves multiple sub-tasks, including semantic understanding, intent recognition, interface structure parsing, and action mapping. Deviations in any of these stages can lead to a final execution result that does not match the user's true intent. Therefore, establishing a reliable "matching degree calculation mechanism" between natural language commands and graphical interface operation sequences has become one of the key technologies for making natural language interaction systems more controllable and trustworthy.

[0004] Existing matching degree calculation methods suffer from the following drawbacks: Uneven granularity: a single step in natural language (e.g., "click button A") may correspond to multiple atomic operations in a graphical interface (e.g., MoveTo + Click). Conversely, graphical interfaces may contain "redundant" operations not mentioned in natural language (e.g., an extra click for "focusing"); Semantic gap: natural language instructions contain rich semantic objects (e.g., "button A," "submit"), while graphical interface operation sequences (especially pure event streams, such as (x, y) coordinates) severely lack semantics. Associating "button A" with the control at coordinates (10, 20) presents a significant challenge for current technologies.

[0005] In view of the above, this application is hereby submitted. Summary of the Invention

[0006] This application aims to provide a graphical interface operation method, apparatus, device, and medium that solves the semantic differences between graphical interface operation sequences and natural language instructions, and robustly handles the mismatch between the granularity and length of the two sequences, thereby improving the accuracy of the calculated matching degree, thus improving the fit between the graphical interface operation results and user expectations, and enhancing the user experience.

[0007] In a first aspect, embodiments of this application provide a graphical user interface operation method, including:

[0008] Receive natural language instructions, and perform intent recognition and parsing on the natural language instructions to obtain a structured intent sequence;

[0009] Candidate graphical interface operation sequences are determined based on the natural language instructions, and parsing and semantic mapping operations are performed on the candidate graphical interface operation sequences to obtain the structured graphical interface operation sequences corresponding to the candidate graphical interface operation sequences.

[0010] A matching score matrix between the structured graphical interface operation sequence and the structured intent sequence is constructed based on a local sequence alignment algorithm. The element values ​​in the matching score matrix represent the local matching scores between the structured graphical interface operation sequence and the structured intent sequence.

[0011] The element with the largest value in the matching score matrix is ​​determined as the matching degree between the structured graphical interface operation sequence and the structured intent sequence;

[0012] If the matching degree is greater than the threshold, then the candidate graphical interface operation sequence is executed.

[0013] According to the technical solution provided in the embodiments of this application, optionally, the structured intent sequence is represented by the following expression:

[0014]

[0015] in, Represents a structured intent sequence. (i=1,2,...,m) represents the intent in the structured intent sequence. (i=1,2,...,m) is a triplet. , Indicates an intentional action. Indicates the intent object, Indicate your intention to input content;

[0016] The structured graphical interface operation sequence is represented by the following expression:

[0017]

[0018] Represents a structured graphical user interface operation sequence. (j=1,2,...,n) represents the atomic operations in the structured graphical interface operation sequence. It is a triplet. , Indicates an operation action. This represents an object that contains control information. This indicates the input content for the operation.

[0019] According to the technical solution provided in the embodiments of this application, optionally, the step of constructing the matching score matrix between the structured graphical interface operation sequence and the structured intent sequence based on the local sequence alignment algorithm includes:

[0020] The values ​​of the elements in the matching score matrix are determined by the following formula:

[0021]

[0022] Represents a structured intent sequence The first i steps and the structured graphical interface operation sequence The local matching score between the first j steps, Let represent the single-step matching score between the i-th step of I and the j-th step of G. and These represent penalty points for open shots.

[0023] According to the technical solution provided in the embodiments of this application, optionally, the single-step matching score... The similarity between the i-th step of I and the j-th step of G is determined based on the weighted similarity in three dimensions: action, object, and input content.

[0024] According to the technical solution provided in the embodiments of this application, optionally, the single-step matching score... Determined by the following formula:

[0025]

[0026] in, For hyperparameter weights, , ) represents the similarity between the i-th step of I and the j-th step of G in the action dimension. Let represent the similarity between the i-th step of I and the j-th step of G in the object dimension. Let I represent the similarity between the i-th step of I and the j-th step of G in terms of the input content dimension.

[0027] According to the technical solution provided in the embodiments of this application, optionally, the similarity in the action dimension... It is determined in the following way:

[0028] The i-th step of I is obtained through a pre-trained semantic embedding model. Intentional action Encode as a first vector;

[0029] The j-th step of G is achieved through a pre-trained semantic embedding model. Operations in Encode as a second vector;

[0030] The similarity between the first vector and the second vector is determined as the similarity in the action dimension. ).

[0031] According to the technical solution provided in the embodiments of this application, optionally, the step of performing parsing and semantic mapping operations on the candidate graphical interface operation sequence to obtain the structured graphical interface operation sequence corresponding to the candidate graphical interface operation sequence includes:

[0032] By combining screen snapshots or control trees, the semantic identifiers of the controls corresponding to the atomic operations in the candidate graphical interface operation sequence are determined, and the semantic identifiers of the controls are associated with the atomic operations to obtain a structured graphical interface operation sequence.

[0033] Secondly, embodiments of this application also provide a graphical user interface operating device, including:

[0034] A preprocessing module is used to receive natural language instructions, perform intent recognition and parsing on the natural language instructions to obtain a structured intent sequence; and determine candidate graphical interface operation sequences based on the natural language instructions, and perform parsing and mapping operations on the candidate graphical interface operation sequences to obtain the structured graphical interface operation sequence corresponding to the candidate graphical interface operation sequence.

[0035] The matching degree calculation module is used to construct a matching score matrix between the structured graphical interface operation sequence and the structured intent sequence based on a local sequence alignment algorithm. The element values ​​in the matching score matrix represent the local matching scores between the structured graphical interface operation sequence and the structured intent sequence. The element value with the largest value in the matching score matrix is ​​determined as the matching degree between the structured graphical interface operation sequence and the structured intent sequence.

[0036] An execution module is configured to execute the candidate graphical interface operation sequence if the matching degree is greater than a threshold.

[0037] Thirdly, embodiments of this application also provide an electronic device, the electronic device comprising:

[0038] Processor and memory;

[0039] The processor executes the steps of the graphical interface operation method as described in any embodiment by calling the program or instructions stored in the memory.

[0040] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a program or instructions that cause a computer to perform the steps of the graphical interface operation method as described in any embodiment.

[0041] In summary, this application proposes a graphical user interface (GUI) operation method. By parsing and semantically mapping candidate GUI operation sequences, semantic information is added to the GUI operation sequences, resolving the semantic differences between GUI operation sequences and natural language commands. Furthermore, by processing natural language commands into structured intent sequences and candidate GUI operation sequences into structured GUI operation sequences, and based on this, a matching score matrix between the structured GUI operation sequences and the structured intent sequences is constructed using a local sequence alignment algorithm. This robustly addresses the granularity and length mismatch between the two sequences, improving the accuracy of the calculated matching degree. Ultimately, this enhances the alignment between the GUI operation results and user expectations, improving the user experience. Attached Figure Description

[0042] Figure 1 This is a flowchart of a graphical interface operation method provided in an embodiment of this application;

[0043] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0044] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0045] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0046] Figure 1 This is a flowchart illustrating a graphical user interface operation method provided in an embodiment of this application. See also... Figure 1 The graphical interface operation method specifically includes the following steps:

[0047] S110. Receive natural language instructions, perform intent recognition and parsing on the natural language instructions, and obtain a structured intent sequence.

[0048] Natural language instructions refer to the operational intentions of the graphical interface expressed through natural language, such as "Open the settings page and enable dark mode", or "Please click control A", or "Click button A, enter XXX in the pop-up input box, and then click button B to submit", etc.

[0049] Intent recognition and parsing of natural language instructions specifically refers to performing lexical analysis, syntactic analysis, and intent recognition on natural language instructions to obtain intermediate semantic representations, such as actions and targets.

[0050] Intent recognition and parsing of natural language commands to obtain structured intent sequences can be achieved using existing large language models or through keyword matching. For example, keywords in commonly used graphical interface operation terms (such as single point, double click, input, icon, control, etc.) can be collected, and then it can be determined whether a certain pre-collected keyword exists in the natural language command, thereby identifying the action, target control, etc.

[0051] The specific form of a structured intent sequence can be:

[0052] in, Represents a structured intent sequence. (i=1,2,...,m) represents the i-th intent in the structured intent sequence. (i=1,2,...,m) is a triplet. , Indicates an intentional action (e.g., clicking). Represents the intent object (e.g., control A). Indicates the intended input content (e.g., specifically a login account).

[0053] S120. Determine candidate graphical interface operation sequences based on natural language instructions, and perform parsing and semantic mapping operations on the candidate graphical interface operation sequences to obtain the structured graphical interface operation sequences corresponding to the candidate graphical interface operation sequences.

[0054] Among them, determining the candidate graphical interface operation sequence based on natural language instructions can be achieved through existing large models. For example, natural language instructions and graphical interface images can be input into a visual large model, and prompts can guide the visual large model to generate candidate graphical interface operation sequences.

[0055] For example, if a user's natural language instruction is: "Click button A, enter XXX in the pop-up input box, and then click button B to submit," the corresponding candidate graphical interface operation sequence is: MoveTo(10,20)->DoubleClick()->MoveTo(30,40)->Click()->Type("XXX")->MoveTo(50,60)->Click(). MoveTo(10,20) means "move to coordinates (10,20)," where (10,20) represents the position of button A. DoubleClick() means "double-click," MoveTo(30,40) means "move to coordinates (10,20)," Click() means "click," and Type("XXX") means "enter content XXX."

[0056] The process involves parsing and semantic mapping candidate graphical interface (GUI) operation sequences to obtain structured GUI operation sequences. This includes: determining the semantic identifiers of controls corresponding to atomic operations in the candidate GUI operation sequences by combining screen snapshots or control trees, and associating these semantic identifiers with the atomic operations to obtain structured GUI operation sequences. Specifically, each atomic operation is associated with the GUI context (such as a screen snapshot or control tree) at the time of its execution. By using the screen snapshot or control tree corresponding to the atomic operation, the actual control on which the atomic operation is applied can be identified. The goal is to map atomic operations to specific controls; for example, mapping the atomic operation "Click(10,20)" to control A, where (10,20) represents control A, thus adding semantic information to the atomic operations.

[0057] The structured graphical interface operation sequence is represented by the following expression:

[0058]

[0059] Represents a structured graphical user interface operation sequence. (j=1,2,...,n) represents the atomic operations in the structured graphical interface operation sequence. It is a triplet. , Indicates an operation action. This represents an object that contains control information. This indicates the input content for the operation.

[0060] For example, one step in the candidate graphical interface operation sequence is "Click(10,20)", which can be converted into one step in the structured graphical interface operation sequence as follows: ,in, It is an object that contains information about the controls. ,Right now It records relevant information about control A, and maps the coordinates (10,20) to control A to give it semantic information.

[0061] S130. Construct a matching score matrix between the structured graphical interface operation sequence and the structured intent sequence based on the local sequence alignment algorithm. The element values ​​in the matching score matrix represent the local matching scores between the structured graphical interface operation sequence and the structured intent sequence.

[0062] For example, using the Smith-Waterman (SW) local sequence alignment algorithm, a... Score matrix .

[0063] express The former Steps and The former The best local matching score between each step.

[0064] For example, the values ​​of the elements in the matching score matrix are determined by the following calculation formula:

[0065]

[0066] Represents a structured intent sequence The first i steps and the structured graphical interface operation sequence The local matching score between the first j steps, Let represent the single-step matching score between the i-th step of I and the j-th step of G. and These represent the penalty points for empty spaces, which can be pre-set empirical values. max() represents a function that takes the maximum value.

[0067] in, , , All are 0. The i-th step of I corresponds to the j-th step of G, meaning the operations in these two steps are corresponding. For example, if the i-th step of I is "Click button A", and the j-th step of G is "Click(10,20)", where (10,20) represents the position of button A, then the steps correspond. Please confirm.

[0068] The case where step i in the intent sequence has no corresponding step in the operation sequence means that step i in the intent sequence has no corresponding step in the operation sequence. For example, if a user's intent is "click button A, then click button B", where clicking button B corresponds to step i in the intent sequence, but operation sequence G performs a scrolling operation after clicking button A, this is a case where step i in the intent sequence has no corresponding step in the operation sequence. To allow the matching operation to continue, step i in the intent sequence is ignored, i.e., "clicking button B" is ignored, but this will result in a deduction of points. .

[0069] The j-th step in G has no corresponding step in I, meaning that the j-th step of the operation sequence has no corresponding description in the intention sequence, or that the intention sequence is missing this step, which is equivalent to skipping this step in the operation sequence, and therefore points will be deducted. For example, the user's intent is "enter username", but before performing "enter", the operation sequence first performs a "click input box" focusing action. This "click input box" focusing action is a necessary atomic operation, but it is omitted in the natural language instruction. This "click input box" focusing action is a redundant step in the operation sequence.

[0070] Resetting and starting a new local match means abandoning further matching and alignment when the sequence alignment is unsatisfactory, and instead starting from the current position to try to find a new, better-matching local segment. For example, if the cumulative score of the alignment matching results in the first 5 steps is already negative (indicating that the first 5 steps were poorly matched), continuing to iterate the matching will only make the score lower. In this case, you can choose to start over from the current position.

[0071] By combining the Smith-Waterman (SW) local sequence alignment algorithm with the setting of gap penalties, this method can effectively "absorb" and "skip" redundant operations (such as focus and scrolling) that are common in graphical interface sequences but not mentioned in natural language, greatly improving the robustness of matching real operation sequences.

[0072] This represents the single-step matching score between the i-th step of I and the j-th step of G, which is the core improvement of this embodiment. The single-step matching score in this embodiment... The similarity between the i-th step of I and the j-th step of G is determined based on the weighted similarity in three dimensions: action, object, and input content.

[0073] For example, single-step matching score Determined by the following formula:

[0074]

[0075] in, For hyperparameter weights, , ) represents the similarity between the i-th step of I and the j-th step of G in the action dimension. Let represent the similarity between the i-th step of I and the j-th step of G in the object dimension. Let I represent the similarity between the i-th step of I and the j-th step of G in terms of the input content dimension.

[0076] Calculate similarity along the action dimension The purpose is to compare the matching between "intended actions" (such as "click") in natural language and "physical actions" (such as "Click") in a graphical interface.

[0077] Specifically, similarity in the action dimension It can be determined in the following way:

[0078] By using pre-trained semantic embedding models (such as BERT, Word2Vec) to embed the i-th step of I... Intentional action Encode as the first vector; use a pre-trained semantic embedding model to embed G at step j. Operations in Encode it as a second vector; determine the similarity between the first and second vectors (e.g., cosine similarity, Euclidean distance) as the similarity in the action dimension. For example, the first vector. The second vector , .in, Represents the encoding function. This represents the function for calculating cosine similarity.

[0079] Similarity in the object dimension It is a measure of cross-modal semantic alignment, the purpose of which is to compare "intent objects" (such as "button A") in natural language with "control entities" in a graphical interface. This is to bridge the "semantic gap" between natural language instructions and graphical interface operation sequences. A specific implementation could be using semantic embedding to perform cross-modal comparisons between natural language descriptions and multiple attributes of graphical interface controls. Specifically, this involves obtaining the embedding of the natural language intent object. Get graphical interface controls Descriptive embedding of multiple attributes: (e.g., "A") (e.g., "Button"), (e.g., "btn_A"), where, , , These are all descriptive embeddings of specific attributes, and other attributes can also be extended.

[0080] It can be With all The maximum similarity between attribute embeddings, or With all The weighted average of similarity among attribute embeddings.

[0081] For example, it can be represented by the following expression. :

[0082]

[0083] The advantage of this setup is its extreme flexibility; the "button A" command in natural language can be matched simultaneously with... ("A") and ("Button"), thus obtaining a higher score. The "Submit Button" in natural language instructions can be matched... The "Submit" function enables the algorithm to handle synonyms, thus making it more adaptable and robust. By utilizing semantic embedding technology, cross-modal matching of the "intent object" in natural language instructions with the "control attributes" in the graphical interface was successfully achieved, solving the biggest semantic gap problem mentioned in the background technology.

[0084] Similarity in the input content dimension Additional values ​​used for comparison operations (e.g., the input content XXX). Similarity It can be the similarity between content strings, if and If all fields are empty (e.g., a Click operation where only the user clicks and no content is entered), then... If neither is empty, then both (referring to...) can be... The Jaccard similarity of ) was determined to be Or calculate first The edit distance, also known as the Levenshtein distance, refers to the minimum number of single-character edit operations required to transform one string into another. The edit distance is then normalized, and the normalized value is determined as follows: If one is empty and the other is not empty, then (or a lower penalty value).

[0085] The scoring matrix is ​​calculated hierarchically (Action, Object, Value) and step-by-step. When the matching degree is low, it is possible to clearly trace back to... ), still If a problem occurs, it will be easier to debug and optimize.

[0086] This is a weight used to balance the importance of matching at different levels. Its specific value can be set based on experience; typically, β is greater than α and β is greater than γ. This is because in natural language control interface scenarios, the "object being operated on" (e.g., which button to click) is usually more important than the "action itself" (e.g., whether it's a single click or a double click). If the wrong object is found, even if the action is correct, it's useless. Therefore, similarity at the object dimension is crucial. The weight β is usually the highest.

[0087] In some implementations, it is assumed that natural language instruction I includes: (Click, Button A, null)

[0088] : (input, input box, "XXX"), (Click, Button B / Submit, null). The graphical interface sequence G includes: : (Click, W1{Type:Button, Text:"A"}, null), (Click, W2{Type:TextBox,Text:""}, null) This step is a redundant focus click. : (Input, W2{Type:TextBox, Text:""}, "XXX"), : (Click, W3{Type:Button, Text:"Submit"}, null).

[0089] The single-step matching score of I at step i and G at step j The calculation process is as follows:

[0090] Comparing (Click, Button A) with (Click, W1{Type:Button, Text:"A"}, null), since "Button A" matches both "Button" and "A", the similarity is higher in the action dimension. Similarity in the object dimension The scores are all relatively high, resulting in a high final single-step matching score. The score is high, therefore it matches elements in the score matrix. Relatively high.

[0091] Comparing (Click, Button A) with (Click, W2{Type:TextBox, Text:""}, null), "Click" matches "Click", but "Button A" does not match "W2{Type:TextBox, Text:""}". Therefore, the similarity is only in the action dimension. The similarity is relatively high in terms of object dimensions. The scores were all low, resulting in a low final single-step matching score. It won't be too high.

[0092] The Smith-Waterman (SW) local sequence alignment algorithm is used to calculate the elements in the matching score matrix. Sometimes, you will find (Low score) is not as good as (i.e., the score when skipping G2), at which point the algorithm will tend to find the optimal alignment path: .because , , All are very high, and It was only considered a small open penalty. Therefore, the final matching degree will be high, which is in line with expectations.

[0093] S140. The element with the largest value in the matching score matrix is ​​determined as the matching degree between the structured graphical interface operation sequence and the structured intent sequence.

[0094] S150. If the matching degree is greater than the threshold, then execute the candidate graphical interface operation sequence.

[0095] The graphical user interface (GUI) operation method provided in this application resolves the semantic differences between GUI operation sequences and natural language commands by parsing and semantically mapping candidate GUI operation sequences. It adds semantic information to these sequences through parsing and semantic mapping, thus addressing the semantic discrepancies. Furthermore, by processing natural language commands into structured intent sequences and candidate GUI operation sequences into structured GUI operation sequences, and then combining this with a local sequence alignment algorithm to construct a matching score matrix between the structured GUI operation sequences and the structured intent sequences, it robustly handles the granularity and length mismatch between the two sequences. This improves the accuracy of the calculated matching degree, thereby enhancing the alignment between the GUI operation results and user expectations, and improving the user experience.

[0096] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides a graphical user interface (GUI) operation device, comprising: a preprocessing module, configured to receive natural language instructions, and perform intent recognition and parsing on the natural language instructions to obtain a structured intent sequence; and determine candidate GUI operation sequences according to the natural language instructions, and perform parsing and mapping operations on the candidate GUI operation sequences to obtain a structured GUI operation sequence corresponding to the candidate GUI operation sequence; a matching degree calculation module, configured to construct a matching score matrix between the structured GUI operation sequence and the structured intent sequence based on a local sequence alignment algorithm, wherein the element values ​​in the matching score matrix represent the local matching scores of the structured GUI operation sequence and the structured intent sequence; and determine the element value with the largest value in the matching score matrix as the matching degree between the structured GUI operation sequence and the structured intent sequence; and an execution module, configured to execute the candidate GUI operation sequence if the matching degree is greater than a threshold.

[0097] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0098] The apparatus of the above embodiments is used to implement the corresponding graphical interface operation method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0099] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 2 As shown, the electronic device 500 includes one or more processors 501 and memory 502.

[0100] The processor 501 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 500 to perform desired functions.

[0101] The memory 502 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 501 may execute the program instructions to implement the graphical interface operation method and / or other desired functions described in any embodiment of this application above. Various contents such as initial external parameters and thresholds may also be stored in the computer-readable storage medium.

[0102] In one example, the electronic device 500 may further include an input device 503 and an output device 504, these components being interconnected via a bus system and / or other forms of connection mechanisms (not shown). The input device 503 may include, for example, a keyboard, a mouse, etc. The output device 504 may output various information to the outside, including warning messages, braking force, etc. The output device 504 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0103] Of course, for the sake of simplicity, Figure 2 Only some of the components of the electronic device 500 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 500 may include any other suitable components depending on the specific application.

[0104] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps of the graphical interface operation method provided in any embodiment of this application.

[0105] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0106] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps of the graphical interface operation method provided in any embodiment of this application.

[0107] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0108] It should be noted that the terminology used in this application is for the purpose of describing specific embodiments only and is not intended to limit the scope of this application. As shown in the specification and claims of this application, unless the context clearly indicates otherwise, words such as "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, or apparatus. Without further limitations, an element defined by the phrase "comprising an..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element.

[0109] It should also be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this application. Unless otherwise expressly specified and limited, the terms "installed," "connected," "linked," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication between two elements. For those skilled in the art, the specific meaning of the above terms in this application can be understood according to the specific circumstances.

[0110] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. The above descriptions are only preferred embodiments of this application. It should be noted that due to the limitations of written expression, while there are objectively infinite specific structures, those skilled in the art can make several improvements, modifications, or changes without departing from the principles of this invention, and can also combine the above technical features in an appropriate manner. These improvements, modifications, changes, or combinations, or the direct application of the inventive concept and technical solution to other situations without modification, should all be considered within the scope of protection of this application.

Claims

1. A graphical interface operation method, characterized in that, include: Receive natural language instructions, and perform intent recognition and parsing on the natural language instructions to obtain a structured intent sequence; Candidate graphical interface operation sequences are determined based on the natural language instructions, and parsing and semantic mapping operations are performed on the candidate graphical interface operation sequences to obtain the structured graphical interface operation sequences corresponding to the candidate graphical interface operation sequences. A matching score matrix between the structured graphical interface operation sequence and the structured intent sequence is constructed based on a local sequence alignment algorithm. The element values ​​in the matching score matrix represent the local matching scores between the structured graphical interface operation sequence and the structured intent sequence. The element with the largest value in the matching score matrix is ​​determined as the matching degree between the structured graphical interface operation sequence and the structured intent sequence; If the matching degree is greater than the threshold, then the candidate graphical interface operation sequence is executed.

2. The graphical interface operation method according to claim 1, characterized in that, The structured intent sequence is represented by the following expression: in, Represents a structured intent sequence. (i=1,2,...,m) represents the intents in the structured intent sequence. (i=1,2,...,m) is a triplet. , Indicates an intentional action. Indicates the intent object, Indicate your intention to input content; The structured graphical interface operation sequence is represented by the following expression: Represents a structured graphical user interface operation sequence. (j=1,2,...,n) represents the atomic operations in the structured graphical interface operation sequence. It is a triplet. , Indicates an operation action. This represents an object that contains control information. This indicates the input content for the operation.

3. The graphical interface operation method according to claim 2, characterized in that, The step of constructing a matching score matrix between the structured graphical interface operation sequence and the structured intent sequence based on a local sequence alignment algorithm includes: The values ​​of the elements in the matching score matrix are determined by the following formula: Represents a structured intent sequence The first i steps and the structured graphical interface operation sequence The local matching score between the first j steps, Let I represent the single-step matching score between the i-th step of I and the j-th step of G; and These represent penalty points for open shots.

4. The graphical interface operation method according to claim 3, characterized in that, The single-step matching score The similarity between the i-th step of I and the j-th step of G is determined based on the weighted similarity in three dimensions: action, object, and input content.

5. The graphical interface operation method according to claim 4, characterized in that, The single-step matching score Determined by the following formula: in, For hyperparameter weights, , ) represents the similarity between the i-th step of I and the j-th step of G in the action dimension. Let represent the similarity between the i-th step of I and the j-th step of G in the object dimension. Let I represent the similarity between the i-th step of I and the j-th step of G in terms of the input content dimension.

6. The graphical interface operation method according to claim 5, characterized in that, Similarity in the action dimension It is determined in the following way: The i-th step of I is obtained through a pre-trained semantic embedding model. Intentional action Encode as a first vector; The j-th step of G is achieved through a pre-trained semantic embedding model. Operations in Encode as a second vector; The similarity between the first vector and the second vector is determined as the similarity in the action dimension. ).

7. The graphical interface operation method according to claim 1, characterized in that, The step of parsing and semantically mapping the candidate graphical interface operation sequences to obtain the structured graphical interface operation sequences corresponding to the candidate graphical interface operation sequences includes: By combining screen snapshots or control trees, the semantic identifiers of the controls corresponding to the atomic operations in the candidate graphical interface operation sequence are determined, and the semantic identifiers of the controls are associated with the atomic operations to obtain a structured graphical interface operation sequence.

8. A graphical user interface operating device, characterized in that, include: A preprocessing module is used to receive natural language instructions, and to perform intent recognition and parsing on the natural language instructions to obtain a structured intent sequence; And determine candidate graphical interface operation sequences based on the natural language instructions, and perform parsing and mapping operations on the candidate graphical interface operation sequences to obtain the structured graphical interface operation sequences corresponding to the candidate graphical interface operation sequences; The matching score calculation module is used to construct a matching score matrix between the structured graphical interface operation sequence and the structured intent sequence based on a local sequence alignment algorithm. The element values ​​in the matching score matrix represent the local matching scores between the structured graphical interface operation sequence and the structured intent sequence. The element with the largest value in the matching score matrix is ​​determined as the matching degree between the structured graphical interface operation sequence and the structured intent sequence; An execution module is configured to execute the candidate graphical interface operation sequence if the matching degree is greater than a threshold.

9. An electronic device, characterized in that, The electronic device includes: Processor and memory; The processor executes the steps of the graphical interface operation method as described in any one of claims 1 to 7 by calling the program or instructions stored in the memory.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program or instructions that cause a computer to perform the steps of the graphical user interface operation method as described in any one of claims 1 to 7.

Images