Cross-software operation agent training method based on structural perception and few-shot learning

Abstract

The application relates to a cross-software operation intelligent agent training method based on structure perception and few-shot learning, which comprises the following steps: receiving the operation intention of a user on a source software, carrying out pretreatment, and carrying out intention coding to obtain a high-dimensional intention semantic vector; extracting control attribute information of a software GUI on the source software, constructing a control tree, and converting the control tree into a high-dimensional structure semantic vector; capturing a GUI screen on the source software and carrying out image coding to obtain a visual feature vector; adopting a cross-attention mechanism to align and fuse the high-dimensional intention semantic vector, the high-dimensional structure semantic vector and the visual feature vector in semantic consistency of attention, so as to obtain a fused feature representation; based on the fused feature representation, a prototype library is constructed, a small amount of user operation samples on a target software are obtained, and the prototype library is updated by using the small amount of user operation samples, so as to realize knowledge transfer and complete a training process. Compared with the prior art, the application has the advantages of strong generalization ability, high efficiency and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a cross-software operation agent training method based on structure perception and few-shot learning. BACKGROUND

[0002] Existing graphical user interface (GUI) operation agent systems mainly rely on visual input (such as screenshots), identify interface elements through computer vision models, and predict possible user operation paths. This method usually uses convolutional neural networks (CNN) or text-vision multi-modal models based on transformers (Transformer) for perception, and uses reinforcement learning or behavior cloning algorithms for decision modeling.

[0003] However, in practical applications, due to the large differences in control layout, style, platform standards, etc. between different software, pure visual-text multi-modal models often fail to achieve good cross-software migration. Even if the interfaces have high similarity in function and logic, the operation learning method relying only on visual features will fail. When facing a new software or a version with slightly changed interface structure, it often needs to collect a large amount of labeled data from scratch and retrain the model, resulting in poor system generalization ability and high development and deployment cost.

[0004] In addition, the current mainstream method generally ignores the semantic structure and hierarchical information behind the GUI, such as control tree, DOM tree, and other structural information, which contains the organizational logic and interaction relationship between interface elements. Different software relies on these structural information to establish logical relationships. If these structural information and visual-text input can be jointly modeled, not only the operation agent's understanding of the interface semantics can be improved, but also the robustness and generalization ability of the model when migrating between software can be improved. Therefore, there is an urgent need for an operation system that can integrate GUI structural information and user intent and has few-shot learning ability to achieve more efficient interface operation agents.

[0005] In summary, the current GUI operation agent faces three core problems in practical applications: first, the lack of understanding of interface structure, which makes the model unable to accurately capture the semantic and hierarchical relationships between controls; second, the excessive reliance on visual features, which has poor generalization ability when the software style changes or the platform migrates; third, the high dependence on large-scale training samples, and the lack of efficient few-shot adaptation mechanism. These problems together limit the practicality and expandability of current methods in a diversified software environment, and an innovative method that integrates multi-modal information, structural understanding ability, and few-shot learning ability is urgently needed to improve the overall performance of the system. SUMMARY

[0006] The application aims to provide a cross-software operation agent training method based on structure perception and few-shot learning with low sample dependence.

[0007] The application aims to provide a cross-software operation agent training method based on structure perception and few-shot learning with low sample dependence.

[0008] A cross-software operation agent training method based on structure perception and few-shot learning comprises the following steps:

[0009] User task receiving and intention encoding: receiving the operation intention of the user on the source software, preprocessing, and encoding the intention by using a context-aware text encoding model to obtain a high-dimensional intention semantic vector;

[0010] GUI structure extraction and structure encoding: extracting the control attribute information of the software GUI on the source software, constructing a control tree, and encoding the control tree into a high-dimensional structure semantic vector;

[0011] GUI screen capture and image encoding: capturing the GUI screen on the source software and encoding the image to obtain a visual feature vector;

[0012] Multi-modal information alignment and fusion: using a cross-attention mechanism to align and fuse the high-dimensional intention semantic vector, high-dimensional structure semantic vector, and visual feature vector in semantic consistency of attention to obtain a fused feature representation;

[0013] Few-shot adaptive learning: based on the fused feature representation, constructing a prototype library, obtaining a few user operation samples on the target software, and updating the prototype library by using the few user operation samples to realize knowledge transfer and complete the training process.

[0014] Further, the operation intention of the user includes one or more of natural language instructions and user operation demonstration sequences.

[0015] Further, the control attribute information of the software GUI is extracted through a UI automation interface, wherein the UI automation interface includes Windows UI Automation and Android Accessibility API.

[0016] Further, the control attribute information includes control name, type, parent-child relationship, and screen coordinates, and the step of constructing the control tree comprises:

[0017] Determining the control hierarchical relationship according to the parent-child relationship and determining the root control as the starting point of traversal;

[0018] Recursively adding child nodes according to the control hierarchical relationship to construct a tree structure as the control tree.

[0019] Further, the control tree is encoded and converted by using a graph neural network or a tree network, and the specific steps of using the graph neural network include:

[0020] Each node in the control tree is converted into a graph node, and the control hierarchy relationship in the control tree is converted into an edge to build a graph structure;

[0021] Each graph node is encoded into a feature vector, mapped to a high-dimensional space through linear mapping, and the information of the neighbor graph node is aggregated to update the embedding of the self graph node;

[0022] All graph node embeddings are aggregated into a global representation of the graph as a high-dimensional structural semantic vector;

[0023] The specific steps of using the tree network include:

[0024] The nodes in the control tree are encoded into feature vectors, mapped to a high-dimensional space, and the embedding of the self node is updated according to the child node information, and finally the embedding of the root node is obtained as a high-dimensional structural semantic vector.

[0025] Further, the steps of the multi-modal information alignment and fusion include the following:

[0026] 1) Initialization:

[0027] The high-dimensional intent semantic vector is taken as the query Q, and the high-dimensional structural semantic vector is split into key and value , and the visual feature vector is split into key and value ;

[0028] 2) Intent-structure multi-cross attention alignment:

[0029] Q, , are respectively divided into h subspaces, and each subspace generates a corresponding subspace query, key and value through independent linear transformation, respectively represented as ,…, ,…, , , …, ,…, , , …, ,…, ;

[0030] For each attention head i , the similarity matrix of the query and the key is calculated, and the calculation expression is:

[0031] ,

[0032] In the formula, is a similarity matrix, indicating the association strength of each language word and each control node, is the subspace dimension;

[0033] The similarity matrix is normalized by Softmax to obtain the attention weight, and is fused with the value to obtain the output of the attention head i , The output of the attention head is expressed as:

[0034] ,

[0035] The outputs of all attention heads are spliced and fused to obtain the structure-aligned intent representation , Where the splicing operation expression is:

[0036] ,

[0037] In the formula, is the structure-aligned intent representation, is the splicing operation;

[0038] 3) Structure-visual multi-head cross attention alignment:

[0039] The structure-aligned intent representation is taken as a new query , And , , is divided into h subspaces respectively, and each subspace generates a corresponding subspace query, key and value through an independent linear transformation, respectively expressed as ,…, ,…, , ,…, ,…, , ,…, ,…, ;

[0040] For each attention head i , the similarity matrix of the query and the key is calculated, and the calculation expression is:

[0041] ,

[0042] In the formula, ​​​is a similarity matrix, representing the association strength of each structure node and visual feature;

[0043] is normalized by Softmax to obtain attention weights, and is fused with value to obtain the output of the attention head , denoted as: i

[0044]

[0045] The outputs of all attention heads are spliced and fused to obtain the structure representation after visual verification , which is the fused feature representation, and the splicing operation expression is:

[0046]

[0047] In the formula, is the structure representation after visual verification, that is, the fused feature representation.

[0048] Further, the specific steps of the few-shot adaptive learning include:

[0049] Pre-training phase:

[0050] Based on the fused feature representation, all fused feature representations under each operation category are clustered according to the operation category, the mean value is calculated, the category center of each operation category, that is, the prototype, is obtained, and a prototype library is formed, and the calculation expression of the category center is:

[0051]

[0052] In the formula, is the category center of the operation category , is the sample number of the operation category , and is the fused feature representation of the first sample under the operation category i

[0053] Migration phase:

[0054] Based on a small number of user operation samples on the target software, multi-modal feature extraction and fusion are performed to obtain the fused feature representation of the target software;

[0055] For each operation category , the fused feature representation of the target software is fused with the corresponding prototype on the source software to obtain an updated prototype ​​​​​​​​​​And further get the updated prototype library.

[0056] Further, the few-shot adaptive learning step further comprises the following:

[0057] Obtaining a new task of a user on the target software, extracting a feature vector of the new task and calculating a similarity with the updated prototype, and selecting a control sequence with the largest similarity as an operation path of the new task, wherein the similarity calculation expression is:

[0058] ,

[0059] In the formula, is the similarity, is the feature vector of the new task, is the updated prototype.

[0060] Further, it further comprises an agent operation prediction step, specifically comprising the following:

[0061] Based on the operation path of the new task, multi-round reasoning is carried out by using strategy learning to obtain corresponding step-by-step planning execution instructions as predicted operation instructions.

[0062] Further, it further comprises an agent execution feedback step, specifically comprising the following:

[0063] The predicted operation instructions are fed back to the real GUI screen and executed, while the GUI screen response is monitored to determine whether to proceed to the next step or to perform error repair.

[0064] Compared with the prior art, the present application has the following beneficial effects:

[0065] (1) The present application can realize cross-software generalization operation with a small amount of task examples by using cross-attention mechanism for semantic consistency alignment and fusion of high-dimensional intention semantic vector, high-dimensional structure semantic vector and visual feature vector, and using few-shot adaptive learning to complete knowledge transfer, which greatly reduces the dependence on a large amount of training data.

[0066] (2) The present application performs structure-aware interface modeling by using the control attribute information of the software GUI, and constructs a control tree, compared with the traditional method which only relies on visual pixels, this structured modeling method can capture the semantic and logical association between controls. This hierarchical and semantic relationship helps to identify controls that are similar in function but different in appearance across software, and better understand the task logic in the interface, thereby significantly improving the generalization ability and interaction robustness of the agent.

[0067] (3) In the multi-modal information alignment and fusion, the cross attention mechanism is used, through intention→structure alignment (capture functional role)→structure→visual alignment (verify appearance details)→fusion decision, an end-to-end multi-modal understanding link is formed, and semantic alignment and accurate positioning are realized.

[0068] (4) The few-shot learning technology is adopted, the dependence on a large amount of training data is greatly reduced, when facing a new software environment, through similarity matching, a large number of parameters do not need to be adjusted, complex tasks can be quickly generalized and completed, and an efficient, high-adaptive, migratable and low-cost cross-platform GUI operation intelligent agent method is formed. BRIEF DESCRIPTION OF DRAWINGS

[0069] Figure 1 It is a whole process schematic diagram of the method of the application;

[0070] Figure 2 It is a process schematic diagram of multi-modal alignment and fusion of the application;

[0071] Figure 3 It is a process schematic diagram of the few-shot learning meta-learning method of the application. DETAILED DESCRIPTION

[0072] The application will be described in detail below in combination with the drawings and specific embodiments. The embodiments are implemented on the premise of the technical scheme of the application, detailed implementation modes and specific operation processes are given, but the protection scope of the application is not limited to the following embodiments.

[0073] The embodiment provides a cross-software operation intelligent agent training method based on structure perception and few-shot learning, as shown in the figure, the method comprises the following steps: Figure 1

[0074] S1, user task receiving and intention coding: receiving the operation intention (also called task intention) of a user on a source software, natural language instructions or user operation demonstration forms input into the system can be used as operation intention. For unstructured user operation intention, a natural language processing technology is used to convert into structured data, and a context perception text coding model is used to convert into a high-dimensional intention semantic vector. For example, a user can input a command such as "export to Excel", which is converted into a semantic embedding vector; or a complete operation process (such as clicking "file→export→Excel" in the menu bar) is demonstrated through screen recording, click recording and the like, all intermediate steps are captured and a learnable track is constructed, which is used as the input of the next step.

[0075] ​This step also supports multi-round interaction intent updating and context awareness, improving the understanding ability of complex tasks. The intent encoding converts the user's task intent into a semantic vector representation. For natural language input, BERT, RoBERTa, and other context-aware language models are used to embed sentences into high-dimensional semantic vectors. For example, the instruction "export the table to Excel" will be encoded into an embedding with similar semantics to "save" and "output"; for operation demonstration sequences, RNN or Transformer can be used to model the control path of the time series, capturing the dynamic trend of user interaction behavior. The high-dimensional intent semantic vector output by this step will be one of the key inputs for multi-modal fusion.

[0076] S2, GUI structure extraction and structure encoding: This step extracts the control attribute information in the software interface on the source software in a non-intrusive manner. Through the UI automation interface (such as Windows UI Automation, Android Accessibility API), the attribute information of the software control can be accessed, such as control name, type, parent-child relationship, screen coordinates, etc. By using the UI automation interface as a "bridge", the interface structure analysis can be realized across software without relying on the source code or specific development framework. Secondly, according to the control attribute information received by the UI automation interface, a control tree is constructed. During the construction of the control tree, the control hierarchy relationship is first determined according to the parent-child relationship, and the root control (such as the top-level window) is determined, and then the child nodes are recursively added according to the control hierarchy relationship to construct a tree structure and form a control tree. For example, in a word processing software, this step identifies the complete path "menu bar -> file -> export -> PDF" and its structure representation, laying the foundation for subsequent structure modeling. This step is particularly suitable for third-party software without source code, enhancing the generality of the method.

[0077] Structure encoding is responsible for converting the control tree into a machine-learnable vector representation, using graph neural networks (such as GCN, GAT) or tree networks (such as Tree-LSTM) to encode each level of the control tree into a high-dimensional structure semantic vector. The steps of converting using graph neural networks include:

[0078] Convert each node in the control tree to a graph node, and convert the control hierarchy relationship to an edge to construct a graph structure;

[0079] Encode each graph node into a feature vector, i.e., the graph node is composed of a feature vector of its type (button, input box), text label, state attribute, etc., and then mapped to a high-dimensional space through linear mapping, and aggregate the information of neighboring graph nodes to update the embedding of the graph node itself;

[0080] Aggregate the graph node embedding into a global representation of the graph as a high-dimensional structure semantic vector.

[0081] For example, the tree structure "File -> Export -> PDF" would be represented as a graph consisting of three nodes, where "Export" is a child of "File" and "PDF" is a child of "Export". This structure encodes important cues for the semantic relationship between controls for the system.

[0082] The steps of converting using the tree network include:

[0083] The nodes in the control tree are encoded as feature vectors, mapped to a high-dimensional space, and updated with the embedding of the child nodes, and finally the embedding of the root node is obtained as a high-dimensional structure semantic vector.

[0084] S3, GUI screen capture and image encoding: receiving the GUI screen on the source software and performing image encoding to obtain a visual feature vector.

[0085] S4, multi-modal information alignment and fusion: the high-dimensional intent semantic vector, the high-dimensional structure semantic vector and the visual feature vector are aligned and fused through the cross-attention mechanism. Specifically, the embodiment of the application uses multiple attention heads to cross-model the high-dimensional intent semantic vector, the high-dimensional structure semantic vector and the visual feature vector. The core purpose is to find matching controls according to the structure position and task semantics even if the control semantics or visual appearance are inconsistent. For example, in two programs with different appearances, one button is labeled as "Export" and the other as "Save as...", but they have the same position and structural context, and the embodiment of the application can still match them as the same semantic control. This step significantly enhances the generalization ability of similar functions between different software interfaces.

[0086] As shown in Figure 2 , in the GUI operation agent, the Query (high-dimensional intent semantic vector) needs to extract key information from the Key (high-dimensional structure semantic vector) and the Value (visual feature vector) to accurately match the control and perform the task. Through the calculation formula of cross-attention, the alignment of multiple modal information is realized, and the attention weight matrix is obtained. Multiply the attention weight matrix with the visual feature vector, and this fused vector (fused feature representation) can be used for GUI operation decision-making.

[0087] Specifically, this step includes the following:

[0088] 1) Initialization:

[0089] The high-dimensional intent semantic vector is taken as the query Q, the high-dimensional structure semantic vector is split into the key and the value , and the visual feature vector is split into the key and the value ;

[0090] 2) Intent-structure multi-cross attention alignment:

[0091] Q, , are respectively divided into h subspaces, and each subspace generates a corresponding subspace query, key, and value through an independent linear transformation, respectively denoted as ,…, ,…, , , …, ,…, , , …, ,…, ;

[0092] For each attention head i , the similarity matrix of the query and the key is calculated, and the calculation expression is:

[0093] ,

[0094] where is the similarity matrix, representing the association strength of each language word and each control node, is the subspace dimension;

[0095] The similarity matrix is normalized by Softmax to obtain the attention weight, and the value is fused to obtain the output i of the attention head , denoted as:

[0096] ,

[0097] The outputs of all attention heads are spliced and fused to obtain the structure-aligned intent representation , and the splicing operation expression is:

[0098] ,

[0099] where is the structure-aligned intent representation, is the splicing operation;

[0100] 3) Structure-visual multi-head cross attention alignment:

[0101] The structure-aligned intent representation is taken as a new query , and , 、 are respectively divided into h subspaces, each of which generates a query, a key and a value of the corresponding subspace by an independent linear transformation, respectively denoted as , , , , , , , , ;

[0102] For each attention head i , a similarity matrix of the query and the key is calculated, wherein the calculation expression is:

[0103] ,

[0104] wherein is the similarity matrix, representing the association strength of each structure node and the visual feature;

[0105] The similarity matrix is normalized by Softmax to obtain the attention weight, and is fused with the value to obtain the output i of the attention head , denoted as:

[0106] ,

[0107] The outputs of all attention heads are spliced and fused to obtain the structure representation after visual verification , as the fused feature representation, wherein the splicing operation expression is:

[0108] ,

[0109] wherein is the structure representation after visual verification, that is, the fused feature representation.

[0110] Few-shot adaptive learning: This step quickly adapts to new software tasks through few-shot learning algorithms such as Prototypical Networks. First, build "operation class prototypes" in the source software, such as the "export class operation" prototype, and obtain the center representation by aggregating multiple example vectors. In the target software, only a small number of labeled examples or user demonstrations are needed to find the most similar control sequence to the prototype through feature matching, realizing operation transfer. For example, only one demonstration of "export PDF" can correctly complete the "export JPEG" task in software with similar structure but different styles without retraining the entire model.

[0111] As shown in Figure 3 , this step includes the following:

[0112] Pre-training phase:

[0113] Based on the fused feature representation, cluster all fused feature representations under each operation category, calculate the mean value, and obtain the class center of each operation category, i.e. the prototype, to form a prototype library. The calculation expression of the class center is:

[0114] ,

[0115] where is the class center of operation category , n is the number of samples of operation category , and is the fused feature representation of the th sample under operation category i .

[0116] Transfer phase:

[0117] Based on a small number of user operation samples (such as 1-5 manually labeled or demonstrated operation data) on the target software, perform multi-modal feature extraction and fusion to obtain the fused feature representation of the target software;

[0118] For each operation category , fuse the fused feature representation of the target software with the corresponding prototype on the source software to obtain the updated prototype , and further obtain the updated prototype library;

[0119] Get the new task of the user on the target software, extract the feature vector of the new task and calculate the similarity with the updated prototype, and select the control sequence with the largest similarity as the operation path of the new task. The similarity calculation expression is:

[0120] ,

[0121] wherein, is the similarity, is the feature vector of the new task.

[0122] S6, agent operation prediction: this step outputs the final operation action prediction according to the fused feature representation of the target software, which may include basic GUI operations such as clicking, inputting, sliding, etc. By aligning the candidate control with the intention vector, the target node to be operated and its specific action are predicted. For example, when performing the "create new project" task, not only the "new" button needs to be clicked, but also the subsequent form filling steps and the execution order of the "confirm" operation need to be determined. This step can perform multi-round reasoning through policy learning (such as reinforcement learning) to achieve step-by-step planning and execution of complex tasks.

[0123] S7, agent execution feedback: this step is responsible for feeding back the predicted operation instructions to the real GUI interface and executing the operation, while monitoring the interface response to decide whether to enter the next step or perform error repair. The execution method can use automated control tools (such as PyAutoGUI, Sikuli) to simulate mouse clicks or keyboard inputs, and listen to control changes, popup window states, and other feedback. For example, after clicking "export", if a save path window pops up, the system will recognize the change and continue to complete the path selection and file confirmation and subsequent steps. If the operation fails or the control is unreachable, the system can automatically roll back, re-plan, or request user feedback.

[0124] In summary, the present application systematically solves the core problems of existing GUI operation agents from the aspects of semantic, structural and visual modeling, multi-modal fusion, and few-shot adaptation. The core idea is that: on the one hand, the control tree structure information widely existing in software interfaces is fully mined and utilized, and the semantic understanding ability and control recognition robustness are enhanced through structural perception modeling; on the other hand, a multi-modal joint learning framework is introduced to integrate natural language operation intention description, visual information and structural information, realize semantic alignment and precise positioning; in addition, the present application uses few-shot learning technology to greatly reduce the dependence on a large amount of training data, so that it can still quickly generalize and complete complex tasks when facing new software environments. These ideas cooperatively constitute an efficient, transferable and low-cost cross-platform GUI operation agent method.

[0125] If the above functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or partially contribute to the prior art, or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0126] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0127] The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in one or more flows and / or blocks.

[0128] These computer program instructions can also be stored in a computer readable storage medium that can direct the computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable storage medium produce a manufactured product including instruction apparatus, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in one or more flows and / or blocks.

[0129] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operations steps are performed on the computer or other programmable data processing devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable data processing devices provide the function of realizing the processes specified in the flowcharts Figure 1 one flow or multiple flows and / or the functions specified in one block or multiple blocks. Figure 1 one flow or multiple flows and / or the functions specified in one block or multiple blocks.

[0130] Although the preferred embodiments of the application have been described, those skilled in the art will be able to make additional modifications and variations to these embodiments without departing from the spirit and scope of the application. Accordingly, it is intended that the appended claims be construed to include all such modifications and variations as fall within the scope of the application.

[0131] Obviously, various modifications and changes can be made to the present application by those skilled in the art without departing from the spirit and scope of the present application. Thus, it is intended that the present application encompass such modifications and changes as fall within the scope of the appended claims and their equivalents.

Claims

1. A method for training a cross-software operation agent based on structure perception and few-shot learning, characterized in that, The method comprises the following steps: User task receiving and intention encoding: receiving the operation intention of the user on the source software, preprocessing, and encoding the intention by using a context-aware text encoding model to obtain a high-dimensional intention semantic vector; GUI structure extraction and structure encoding: extracting the control attribute information of the software GUI on the source software, constructing a control tree, and encoding the control tree into a high-dimensional structure semantic vector; GUI screen capture and image encoding: capturing the GUI screen on the source software and performing image encoding to obtain a visual feature vector; Multi-modal information alignment and fusion: using a cross-attention mechanism to align and fuse the high-dimensional intention semantic vector, the high-dimensional structure semantic vector, and the visual feature vector in terms of semantic consistency of attention to obtain a fused feature representation; Few-shot adaptive learning: based on the fused feature representation, constructing a prototype library, obtaining a few user operation samples on the target software, and updating the prototype library by using the few user operation samples to realize knowledge transfer and complete the training process.

2. The method of claim 1, wherein the method is characterized by, The operation intention of the user includes one or more of natural language instructions and user operation demonstration sequences.

3. The method of claim 1, wherein the method further comprises: The control attribute information of the software GUI is extracted through a UI automation interface, wherein the UI automation interface includes a Windows UI Automation and an Android Accessibility API.

4. The method of claim 1, wherein, The control attribute information includes control names, types, parent-child relationships, and screen coordinates, and the step of constructing the control tree comprises: determining the control hierarchical relationship according to the parent-child relationship and determining the root control as the starting point of traversal; recursively adding child nodes according to the control hierarchical relationship to construct a tree structure as the control tree.

5. The method of claim 4, wherein, The control tree is encoded and converted by using a graph neural network or a tree network, and the specific steps of using the graph neural network comprise: each node in the control tree is converted into a graph node, and the control hierarchical relationship in the control tree is converted into an edge to construct a graph structure; each graph node is encoded into a feature vector, mapped to a high-dimensional space, and the information of neighbor graph nodes is aggregated to update the embedding of the self graph node; all graph node embeddings are aggregated into a global representation of the graph as a high-dimensional structure semantic vector; The specific steps of using the tree network comprise: the nodes in the control tree are encoded into feature vectors, mapped to a high-dimensional space, and the embedding of the root node is updated according to the information of the child nodes to obtain a high-dimensional structure semantic vector.

6. The method of claim 1, wherein, The steps of multi-modal information alignment and fusion specifically include the following: 1) initialization; 2) intention-structure multi-cross attention alignment; splitting the high-dimensional intent semantic vector as a query Q, a high-dimensional structure semantic vector into a key and a value , a visual feature vector into a key and a value ; 3) structure-visual multi-head cross attention alignment; Q, , are respectively divided into h subspaces, each of which generates a query, a key, and a value of the corresponding subspace through an independent linear transformation, and are respectively represented as ,…, ,…, , , …, ,…, , , …, ,…, ; For each attention head i , compute a similarity matrix of queries and keys , where the computation expression is: ， In the formula, is a similarity matrix, indicating the association strength of each language word with each control node, is a subspace dimension; Similarity matrix analyzed using Softmax Normalize to obtain attention weights, and then compare them with the values. To achieve fusion and obtain attention i Output , is represented as: ， concatenating the outputs of all attention heads performing splicing fusion to obtain a structure-aligned intent representation wherein the splicing operation expression is: ， In the formula, is a structural alignment intent representation, is a stitching operation; The specific steps of few-shot adaptive learning comprise: aligning the structure of the intent representation as a new query , aligning the structure of the intent representation as a new query , , , respectively, into h subspaces, each of which generates a query, key, value for the corresponding subspace through an independent linear transformation, respectively represented as , , , , , , , , ; For each attention head i , compute a similarity matrix of queries and keys , where the computation expression is: ， In the formula, is a similarity matrix, representing the association strength of each structure node and visual feature; Similarity matrix analyzed using Softmax Normalize to obtain attention weights, and then compare them with the values. To achieve fusion and obtain attention i Output , is represented as: ， concatenating the outputs of all attention heads performing splicing fusion to obtain a structure representation after visual verification , as a fused feature representation, wherein the splicing operation expression is: ， In the formula, is a structure representation after visual verification, i.e., a fused feature representation.

7. The method of claim 1, wherein the method further comprises: Pre-training phase: based on the fused feature representation, clustering all fused feature representations under each operation category according to the operation category, calculating the mean value to obtain the category center of each operation category, i.e., the prototype, to form a prototype library, and the calculation expression of the category center is: Migration phase: ， In the formula, For operation category Category center, For operation category The number of samples, For operation category Next i Feature representation of fused samples; ​ Based on a small number of user operation samples on the target software, multi-modal feature extraction and fusion are performed to obtain a fusion feature representation of the target software; For each operation category fusing the fused feature representation of the target software with the corresponding prototype on the source software to obtain an updated prototype and further obtaining an updated prototype library.

8. The method of claim 7, wherein the method further comprises: The few-shot adaptive learning step further includes the following: A new task of a user on the target software is obtained, a feature vector of the new task is extracted, and a similarity with the updated prototype is calculated, and a control sequence with the largest similarity is selected as an operation path of the new task, wherein the similarity calculation expression is: ， In the formula, is the similarity, is the feature vector of the new task, is the updated prototype.

9. The method of claim 8, wherein, The agent operation prediction step further includes the following: Based on the operation path of the new task, multi-round reasoning is performed using policy learning to obtain corresponding step-by-step planning execution instructions as predicted operation instructions.

10. The method of claim 9, wherein, The agent execution feedback step further includes the following: The predicted operation instructions are fed back to the real GUI screen and executed, and the GUI screen response is monitored to determine whether to proceed to the next step or to perform error repair.

Images