Multi-modal Action Transform model and intelligent task execution method thereof

Abstract

The invention relates to the field of enterprise process automation, in particular to a multi-mode Action Transform model and an intelligent task execution method thereof. The technical problems that in an existing Web page operation automation technology, flexibility and adaptability are insufficient, generalization ability is weak, context understanding ability is limited, multi-modal information integration is insufficient, and data utilization efficiency is low are solved. According to the technical scheme, the multi-modal Action Transform model and the intelligent task execution method thereof comprise a multi-modal input processing module, a multi-modal fusion module, a Transform encoder and an operation prediction module; according to the method, various inputs such as images, texts and DOM structures are processed at the same time, the fault-tolerant ability and adaptability to page changes are improved, the long-range dependency relationship in the operation sequence can be effectively captured and utilized through the attention mechanism, and deep fusion of various modal information such as the images, the texts and the DOM structures is achieved through the attention mechanism and the feature fusion layer.

Description

Technical Field

[0001] The present invention relates to the field of enterprise process automation, and in particular to a multimodal Action Transformer model and an intelligent task execution method thereof. Background Art

[0002] With the rapid development of artificial intelligence technology, especially the emergence of large language models (LLM), AI is increasingly being used in various fields. In the field of enterprise process automation, AI technology is playing an increasingly important role, especially in the automation of web page operations.

[0003] Currently, a variety of technical solutions have been applied to the automation of Web page operations:

[0004] Rule-based automation scripts simulate human operations on web pages by writing specific scripts. For example, Selenium WebDriver is a commonly used tool that can control the browser to perform various operations through code. However, this method requires a lot of manual coding and is difficult to adapt to the complex and changing web environment.

[0005] Image recognition-based automation uses computer vision technology to identify elements on a web page and perform corresponding operations. For example, Sikuli is an open source automation tool that uses image recognition to locate and operate elements on the screen. However, this method is easily affected by changes in page layout and has poor stability.

[0006] DOM-based automation locates and manipulates elements by analyzing the DOM (Document Object Model) structure of a web page. For example, the Puppeteer library is a commonly used tool that provides a high-level API to control Chrome or Chromium browsers. This method is relatively more stable, but still requires a lot of manual configuration and maintenance.

[0007] Machine learning-assisted automation, which applies machine learning techniques to web automation;

[0008] However, the existing Web page operation automation technology still has many defects and shortcomings, which are mainly reflected in the following aspects:

[0009] Lack of flexibility and adaptability:

[0010] 1. Rule-based automation scripts: This method is highly dependent on predefined rules and paths, and is difficult to cope with dynamic changes in Web pages. When the page structure, style, or content changes, the scripts often require a lot of manual modification and maintenance, and cannot automatically adapt to the new environment.

[0011] 2. Image recognition basic automation: Although relatively more flexible, it is still susceptible to changes in the visual appearance of the page; for example, when the page theme, color scheme or layout changes, the system's recognition accuracy will drop significantly;

[0012] 3. DOM structure-based automation: This method is sensitive to changes in page structure. When a website undergoes a major update or reconstruction, the DOM-based positioning method may become completely ineffective and the positioning rules need to be rewritten.

[0013] Weak generalization ability:

[0014] 1. Most existing methods are customized for specific websites or applications and are difficult to migrate between different websites or applications. This results in each new automation task needing to be developed from scratch, greatly increasing development and maintenance costs.

[0015] 2. Even within the same website, automation scripts between different pages or versions are often difficult to reuse and lack universality;

[0016] Limited contextual understanding:

[0017] 1. Existing automation methods usually treat each operation as an independent event and lack a global understanding of the entire operation sequence; this results in the system being unable to utilize the contextual information of previous and subsequent operations to optimize the current operation;

[0018] 2. When dealing with complex multi-step tasks, it is easy to make operational errors or fall into an endless loop due to a lack of understanding of the overall process of the task;

[0019] Insufficient integration of multimodal information:

[0020] 1. Most existing methods rely only on a single type of information (such as plain text, plain images, or DOM structures) and cannot effectively integrate multiple types of information;

[0021] 2. Even some of the more advanced AI models, although they can process multimodal input, often simply splice information from different modalities rather than deeply fuse them, resulting in insufficient information utilization;

[0022] Low data utilization efficiency:

[0023] 1. Traditional automation methods rarely learn and improve from historical operations; each task execution is independent and cannot use accumulated experience data to improve performance;

[0024] 2. Even some machine learning-assisted methods are often limited to specific types of tasks or data and have difficulty making full use of large-scale and diverse operational data;

[0025] Therefore, to address the above problems, a multimodal Action Transformer model and its intelligent task execution method are proposed, which aims to transform complex Web operation tasks into a series of intelligent decision-making problems through deep learning technology, and improve the accuracy and robustness of decisions through multimodal information fusion and context understanding. Summary of the invention

[0026] In order to overcome the problems of insufficient flexibility and adaptability, weak generalization ability, limited context understanding ability, insufficient multimodal information integration and low data utilization efficiency in existing Web page operation automation technology.

[0027] The technical solution of the present invention is: a multimodal Action Transformer model, comprising:

[0028] A multimodal input processing module for simultaneously receiving and processing images, text, and structured data (DOM tree) as input;

[0029] The multimodal fusion module is used to deeply fuse the features of different modalities using a multi-head cross-attention mechanism to establish cross-modal associations;

[0030] Transformer encoder, which consists of a multi-layer Transformer encoder structure, each layer contains a self-attention mechanism and a feed-forward neural network to capture long-range dependencies and understand the context of the entire operation sequence;

[0031] The operation prediction module is used to predict the type and parameters of the next operation based on the output of the Transformer encoder using a multi-layer perceptron, including but not limited to click, input, scroll and other operations and their specific parameters.

[0032] Preferably, a multimodal input processing module simultaneously receives and processes images, texts, and structured data (DOM trees) as inputs; a multimodal fusion module adopts a multi-head cross-attention mechanism to deeply fuse features of different modalities to establish cross-modal associations; a Transformer encoder is used to capture long-range dependencies and understand the context of the entire operation sequence; an operation prediction module uses a multi-layer perceptron based on the output of the Transformer encoder to predict the type and parameters of the next operation, including but not limited to click, input, scroll and other operations and their specific parameters.

[0033] Preferably, the multimodal input processing module includes an image processing submodule, a text processing submodule and a DOM processing submodule; the image processing submodule is used to process the page screenshots using a pre-trained convolutional neural network to extract effective visual features; the text processing submodule is used to use BERT and its variant models to perform word segmentation and embedding generation on the page text content to capture text semantic information; the DOM processing submodule is used to process the DOM tree structure using a graph neural network to generate a structured feature representation to capture the relationship between page elements.

[0034] Preferably, the multimodal fusion module generates a comprehensive feature representation by deeply fusing features from different modalities (image, text, DOM structure) for subsequent operation prediction; the multimodal fusion module includes a feature input layer, a multi-head cross attention mechanism unit and a fusion layer; the feature input layer receives feature vectors from the image processing submodule, the text processing submodule and the DOM processing submodule; the multi-head cross attention mechanism unit allows the model to establish associations between different modalities, and achieves deep fusion of features by calculating the attention weights between features of different modalities; the fusion layer further processes the output of the multi-head cross attention mechanism to generate fused features;

[0035] When the multimodal fusion module is working, the following steps are included:

[0036] S301: Obtain visual features of the page from the image processing submodule, obtain text embedding of the page from the text processing submodule, and obtain structural features of the page from the DOM processing submodule;

[0037] S302: Input the extracted visual features, text embedding and structured features into the feature input layer of the multimodal fusion module;

[0038] S303: Process the features of each modality separately, and calculate an attention weight matrix for each head to represent the degree of association between features of different modalities;

[0039] S304: Perform weighted summation of features through the attention weight matrix to generate the output of each head;

[0040] S305: splicing the outputs of all heads to obtain preliminary fusion features;

[0041] S306: The fusion layer further processes the preliminary fusion features and generates the final fusion features through the fully connected layer and activation function;

[0042] S307: Output the fused features to the Transformer encoder.

[0043] Preferably, the Transformer encoder includes an encoder input layer, a self-attention mechanism layer, a feedforward neural network layer and an encoder output layer; the encoder input layer is used to receive the fused multimodal features as input; the self-attention mechanism layer is used to capture the long-range dependencies in the sequence by calculating the attention weights between each element and other elements, allowing the model to consider the information of all other elements in the sequence when processing each input element; the feedforward neural network layer contains two linear transformations and an activation function, which is used for further processing of the output of the self-attention mechanism to extract higher-level features; the encoder output layer is used to generate encoded feature representations, which will be passed to the subsequent operation prediction module to predict the next operation type and parameters;

[0044] The Transformer encoder includes the following steps when working:

[0045] S401: Input the fused multimodal features into the Transformer encoder;

[0046] S402: For each element in the input sequence, a query vector, a key vector, and a value vector are generated;

[0047] S403: Calculate the dot product of the query vector and all key vectors, and use the softmax function to obtain the attention weight of each element to other elements;

[0048] S404: Use the attention weights to perform weighted summation on the value vector to obtain the output of the self-attention mechanism; the output contains information about all elements in the sequence, and each element is weighted according to its relevance to the current element;

[0049] S405: The output of the self-attention mechanism is passed to the feed-forward neural network, and the feed-forward neural network further extracts and transforms the output of the self-attention mechanism;

[0050] S406: The output of the self-attention mechanism is processed by a feedforward neural network to generate an encoded feature representation including context information of the input sequence, and the feature representation is transmitted to the operation prediction module;

[0051] S407: Repeat S401-S406, where each layer receives the output of the previous layer as input and generates a new encoded feature representation, so that the self-attention mechanism and feedforward neural network are usually stacked with multiple layers to more deeply capture the long-range dependencies and contextual information in the sequence.

[0052] Preferably, the operation prediction module includes an operation type prediction unit and an operation parameter prediction unit; the operation type prediction unit is used to predict the type of the next operation, including click, input, scroll, these operation types are the basic units of Web page operation automation, by predicting these operation types, the model can build a complete operation sequence; the operation parameter prediction unit is used to predict the specific parameters of the operation, including the coordinates of the click and the text content of the input;

[0053] When the operation prediction module is working, the following steps are included:

[0054] S501: The operation prediction module receives the output from the Transformer encoder;

[0055] S502: extracting features related to operation prediction from the output of the Transformer encoder, and extracting features related to operation prediction, the features including historical information of the operation sequence, status information of the current page, and representation of user intention;

[0056] S503: using a multi-layer perceptron to classify the processed features, the model selects the most likely operation type from a predefined set of operation types to predict the type of the next operation;

[0057] S504: predicting the click coordinates for a click operation and predicting the input text content for an input operation through regression task and classification task methods;

[0058] S505: converting the predicted operation type and parameters into standardized operation instructions; the instructions are directly transmitted to the Web automation execution engine, and the execution engine executes the corresponding operation on the actual Web page;

[0059] S506: The execution result is used as feedback to update the state and training of the model; the model optimizes its prediction performance by continuously receiving feedback to better adapt to the dynamic Web environment.

[0060] Preferably, the Action Transformer model includes the following steps when working:

[0061] S1: Data preprocessing: Get the screenshot, text content and DOM structure of the current web page; preprocess these raw data, including image scaling, text segmentation, and DOM tree serialization;

[0062] S2: Feature extraction: The image processing submodule extracts visual features; the text processing submodule generates text embeddings; the DOM processing submodule generates structural features;

[0063] S3: Multimodal fusion: Use multi-head cross-attention mechanism to fuse features of different modalities;

[0064] S4: Transformer encoding: The fused features are processed through a multi-layer Transformer encoder to capture long-range dependencies and contextual information;

[0065] S5: Operation prediction: Based on the output of the encoder, predict the type and parameters of the next operation;

[0066] S6: Structured output: converting prediction results into standardized operational instructions;

[0067] S7: Execution and feedback: Pass the operation instructions to the execution engine; the execution results are used as feedback to update the model status and training.

[0068] Preferably, the training of the Action Transformer model is divided into two stages:

[0069] A1: Pre-training stage: Collect a large amount of diverse Web operation data; construct self-supervised learning tasks, such as operation sequence prediction and cross-modal matching; perform pre-training on large-scale data to learn common Web operation patterns;

[0070] A2: Fine-tuning stage: Collect relevant operational data for specific tasks or fields; use supervised learning and reinforcement learning methods for fine-tuning; optimize the performance of the model on specific tasks.

[0071] A multimodal Action Transformer model intelligent task execution method includes the following steps:

[0072] S801: Data collection and preprocessing: Get a screenshot of the current Web page; extract the text content of the page; get the DOM structure of the page; preprocess the collected data, including image scaling, text segmentation, and DOM tree serialization;

[0073] S802: Feature extraction: Use pre-trained convolutional neural networks to process page screenshots and extract visual features; use BERT and its variants to process page text content and generate text embeddings; use graph neural networks to process DOM tree structures and generate structured features;

[0074] S803: Multimodal feature fusion: The visual features, text embedding and structured features extracted in step S802 are input into the multi-head cross attention layer, and the features of different modalities are deeply fused through the multi-head cross attention mechanism; the fused multimodal feature representation is output;

[0075] S804: Transformer encoding: The fused features are input into a multi-layer Transformer encoder. Each layer of the Transformer encoder contains a self-attention mechanism and a feed-forward neural network. Through multi-layer processing, long-range dependencies are captured and the context of the operation sequence is understood.

[0076] S805: Operation prediction: Based on the output of the Transformer encoder, use a multi-layer perceptron to predict the operation; predict the type of the next operation, including click, input, and scroll; predict the specific parameters of the operation, including click coordinates and input text content;

[0077] S806: Structured output generation: convert the predicted operation type and parameters into a standardized instruction format, and generate a structured operation instruction, including the operation type, target element, and parameters;

[0078] S807: Instruction execution: The generated structured instructions are passed to the Web automation execution engine, and the execution engine performs corresponding operations on the actual Web page according to the instructions.

[0079] Preferably, when performing multimodal information encoding, the following steps are included:

[0080] S901: Obtain a screenshot of the current web page and convert it into an RGB pixel matrix format;

[0081] S902: extract all text content on the page, and perform word segmentation and cleaning to remove irrelevant characters and noise;

[0082] S903: Parse the DOM tree of the web page and serialize it into a node list and adjacency matrix format;

[0083] S904: Use the pre-trained convolutional neural network ResNet to process the page screenshot and extract the visual features of the image, including edges, textures, colors, and more advanced object and scene features;

[0084] S905: Convert the extracted features into vector representation;

[0085] S906: Use BERT and its variant RoBERTa to encode text content and generate text embedding;

[0086] S907: Convert text embedding into vector representation;

[0087] S908: Use graph neural network to process the DOM tree, capture the relationship and attribute information between DOM nodes to generate structured features;

[0088] S909: Input the feature vectors of the image, text, and DOM structure into the multi-head cross attention layer; through the attention mechanism, the model can automatically adjust the weights of different modal features to generate a fused multimodal feature representation;

[0089] S910: Integrate the fused multimodal feature representation into a unified vector, which contains all relevant information from the image, text and DOM structure;

[0090] S911: pass this unified vector to the subsequent Transformer encoder;

[0091] When performing structured vector processing, the following steps are included:

[0092] S1001: using a DOM parser to parse the HTML code of a web page into a DOM tree;

[0093] S1002: For each node in the DOM tree, extract its feature vector, the features including the node's tag name, attributes, and text content; the feature vector is generated by encoding each attribute of the node into a one-hot encoding or an embedding vector, and for the tag name, a predefined tag name embedding matrix is ​​used to map each tag name to a vector representation of a fixed length;

[0094] S1003: Use graph neural network to process the DOM tree to extract the relationship and structural information between nodes. GNN propagates information on the graph so that the representation of each node contains the information of its neighboring nodes. GNN uses the update rule of Graph Convolutional Network. The specific calculation formula is as follows:

[0095]

[0096] Among them, A is the adjacency matrix of the graph, D is the degree matrix, and H (l) is the node feature matrix of the lth layer, W (l) is the weight matrix of the lth layer, σ is the activation function;

[0097] S1004: Encode the operation on the web page into a vector representation of a fixed length by mapping the operation type to an embedding vector;

[0098] S1005: For each operation, extract its parameters and encode them, and the parameters are encoded into a numerical vector and an embedded vector;

[0099] S1006: fusing the feature vector of the Web page element and the feature vector of the operation by a concatenation and addition fusion method to generate a structured vector;

[0100] S1007: normalizing the fused feature vector by normalization and standardization methods to ensure the consistency of its dimension and value range;

[0101] S1008: Input the structured vector into the Transformer encoder and the operation prediction module.

[0102] As a preference, when converting the operation intention into behavior, the specific steps are as follows:

[0103] S1101: input natural language instructions;

[0104] S1102: performing natural language processing operations such as word segmentation, part-of-speech tagging, and named entity recognition on the instruction to extract key information and structure;

[0105] S1103: extracting semantic features of the instruction using the NLP model; inputting the features into the classifier to identify the user's operation intention;

[0106] S1104: extracting information of the target element from the instruction; if the operation requires additional parameters, parsing these parameters from the instruction;

[0107] S1105: extract visual features of the page screenshot using the image processing submodule; process the page text content using the text processing submodule to generate text embedding; process the DOM tree structure of the page using the DOM processing submodule to generate structured features; merge the above features with the semantic features of the instruction to obtain a more comprehensive context understanding;

[0108] S1106: Input the fused features into a multi-layer Transformer encoder to capture long-range dependencies and contextual information;

[0109] S1107: Based on the output of the encoder, use a multi-layer perceptron to predict the type and parameters of the next operation;

[0110] S1108: Convert the predicted operation type and parameters into a standardized instruction format, and optimize the instruction according to the context information;

[0111] S1109: passing the generated structured instruction to the Web automation execution engine to execute the corresponding operation, and the execution engine returns the operation result for updating the state and training of the model;

[0112] Among them, the semantic feature extraction formula in the intention recognition step is as follows:

[0113] Feature Vectors=NLPModel(Preprocessed Text);

[0114] Among them, the intent classification formula in the intent recognition step is as follows:

[0115] Intent = argmax i (w i Feature Vectors+b i );

[0116] Among them, in the parameter parsing step, target element recognition uses named entity recognition technology to extract target elements from instructions; parameter extraction uses regular expressions to parse the parameters required for the operation from the instructions;

[0117] Among them, the feature fusion formula in the multimodal information fusion step is as follows:

[0118] Fused Features=Fusion Layer(Image Features,Text Features,DOMFeatures,Semantic Features);

[0119] Among them, the Transformer encoding formula in the operation decision step is as follows:

[0120] EncodedFeatures=Transformer Encoder(FusedFeatures);

[0121] Among them, the operation prediction formula in the operation decision step is as follows:

[0122] Operation = argmax j (MLP(EncodedFeatures) j );

[0123] Among them, the structured output formula in the instruction generation step is as follows:

[0124] StructuredInstruction=Format(Operation,Parameters);

[0125] The execution result output formula is as follows:

[0126] Result=Execution Engine(StructuredInstruction).

[0127] Beneficial effects of the present invention:

[0128] 1. Compared with the existing problems of insufficient flexibility and adaptability of Web page operation automation technology, the present invention significantly improves the system's adaptability to complex and dynamic Web environments by introducing multimodal input and context integration capabilities; the Action Transformer model can simultaneously process multiple inputs such as images, texts, and DOM structures, so that the system is no longer limited to a single recognition method, thereby greatly improving the fault tolerance and adaptability to page changes;

[0129] 2. Compared with the problem of weak generalization ability of Web page operation automation technology under the existing technology, the present invention greatly improves the generalization ability of the system by adopting a deep learning model based on the Transformer architecture; the ActionTransformer model can learn common operation modes and strategies from a large amount of diverse Web operation data, so that the model can easily cope with new websites, new page layouts, and even new Web applications without retraining or adjustment for each specific scenario;

[0130] 3. Compared with the limited context understanding ability of the existing web page operation automation technology, the Action Transformer model of the present invention can effectively capture and utilize the long-range dependencies in the operation sequence through the attention mechanism, which enables the system to understand the context of the entire operation process, thereby making more intelligent and coherent decisions, effectively solving the problem of insufficient context understanding ability in the existing technology;

[0131] 4. Compared with the problem of insufficient multi-modal information integration in the existing Web page operation automation technology, the Action Transformer model of the present invention realizes the deep integration of multiple modal information such as images, texts, DOM structures, etc. through the attention mechanism and feature fusion layer; this fusion enables the model to fully understand the structure and content of the Web page, so as to make more accurate operation decisions;

[0132] 5. Compared with the problem of low data utilization efficiency in the existing Web page operation automation technology, the present invention can make full use of large-scale historical operation data by designing special pre-training tasks and fine-tuning strategies; this not only improves the learning efficiency of the model, but also enables the model to continuously learn and improve from new operation data, solving the problem of low data utilization efficiency in the existing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0133] Figure 1 Shown is a schematic diagram of the Action Transformer model architecture of the multimodal Action Transformer model of the present invention;

[0134] Figure 2 What is shown is a detailed workflow diagram of the multimodal Action Transformer model of the present invention;

[0135] Figure 3 What is shown is a simplified workflow diagram of the multimodal Action Transformer model of the present invention;

[0136] Figure 4 What is shown is a flowchart of the steps of the multimodal Action Transformer model intelligent task execution method of the present invention. DETAILED DESCRIPTION

[0137] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0138] See also Figure 1-4 The present invention provides an embodiment: a multimodal Action Transformer model and an intelligent task execution method thereof, comprising:

[0139] A multimodal input processing module for simultaneously receiving and processing images, text, and structured data (DOM tree) as input;

[0140] The multimodal fusion module is used to deeply fuse the features of different modalities using a multi-head cross-attention mechanism to establish cross-modal associations;

[0141] Transformer encoder, which consists of a multi-layer Transformer encoder structure, each layer contains a self-attention mechanism and a feed-forward neural network to capture long-range dependencies and understand the context of the entire operation sequence;

[0142] The operation prediction module is used to predict the type and parameters of the next operation based on the output of the Transformer encoder using a multi-layer perceptron, including but not limited to click, input, scroll and other operations and their specific parameters.

[0143] Preferably, a multimodal input processing module simultaneously receives and processes images, texts, and structured data (DOM trees) as inputs; a multimodal fusion module adopts a multi-head cross-attention mechanism to deeply fuse features of different modalities to establish cross-modal associations; a Transformer encoder is used to capture long-range dependencies and understand the context of the entire operation sequence; an operation prediction module uses a multi-layer perceptron based on the output of the Transformer encoder to predict the type and parameters of the next operation, including but not limited to click, input, scroll and other operations and their specific parameters.

[0144] Preferably, the multimodal input processing module includes an image processing submodule, a text processing submodule and a DOM processing submodule; the image processing submodule is used to process the page screenshots using a pre-trained convolutional neural network to extract effective visual features; the text processing submodule is used to use BERT and its variant models to perform word segmentation and embedding generation on the page text content to capture text semantic information; the DOM processing submodule is used to process the DOM tree structure using a graph neural network to generate a structured feature representation to capture the relationship between page elements.

[0145] Preferably, the multimodal fusion module generates a comprehensive feature representation by deeply fusing features from different modalities (image, text, DOM structure) for subsequent operation prediction; the multimodal fusion module includes a feature input layer, a multi-head cross attention mechanism unit and a fusion layer; the feature input layer receives feature vectors from the image processing submodule, the text processing submodule and the DOM processing submodule; the multi-head cross attention mechanism unit allows the model to establish associations between different modalities, and achieves deep fusion of features by calculating the attention weights between features of different modalities; the fusion layer further processes the output of the multi-head cross attention mechanism to generate fused features;

[0146] When the multimodal fusion module is working, the following steps are included:

[0147] S301: Obtain visual features of the page from the image processing submodule, obtain text embedding of the page from the text processing submodule, and obtain structural features of the page from the DOM processing submodule;

[0148] S302: Input the extracted visual features, text embedding and structured features into the feature input layer of the multimodal fusion module;

[0149] S303: Process the features of each modality separately, and calculate an attention weight matrix for each head to represent the degree of association between features of different modalities;

[0150] S304: Perform weighted summation of features through the attention weight matrix to generate the output of each head;

[0151] S305: splicing the outputs of all heads to obtain preliminary fusion features;

[0152] S306: The fusion layer further processes the preliminary fusion features and generates the final fusion features through the fully connected layer and activation function;

[0153] S307: Output the fused features to the Transformer encoder.

[0154] Preferably, the Transformer encoder includes an encoder input layer, a self-attention mechanism layer, a feedforward neural network layer and an encoder output layer; the encoder input layer is used to receive the fused multimodal features as input; the self-attention mechanism layer is used to capture the long-range dependencies in the sequence by calculating the attention weights between each element and other elements, allowing the model to consider the information of all other elements in the sequence when processing each input element; the feedforward neural network layer contains two linear transformations and an activation function, which is used for further processing of the output of the self-attention mechanism to extract higher-level features; the encoder output layer is used to generate encoded feature representations, which will be passed to the subsequent operation prediction module to predict the next operation type and parameters;

[0155] The Transformer encoder includes the following steps when working:

[0156] S401: Input the fused multimodal features into the Transformer encoder;

[0157] S402: For each element in the input sequence, a query vector, a key vector, and a value vector are generated;

[0158] S403: Calculate the dot product of the query vector and all key vectors, and use the softmax function to obtain the attention weight of each element to other elements;

[0159] S404: Use the attention weights to perform weighted summation on the value vector to obtain the output of the self-attention mechanism; the output contains information about all elements in the sequence, and each element is weighted according to its relevance to the current element;

[0160] S405: The output of the self-attention mechanism is passed to the feed-forward neural network, and the feed-forward neural network further extracts and transforms the output of the self-attention mechanism;

[0161] S406: The output of the self-attention mechanism is processed by a feedforward neural network to generate an encoded feature representation including context information of the input sequence, and the feature representation is transmitted to the operation prediction module;

[0162] S407: Repeat S401-S406, where each layer receives the output of the previous layer as input and generates a new encoded feature representation, so that the self-attention mechanism and feedforward neural network are usually stacked with multiple layers to more deeply capture the long-range dependencies and contextual information in the sequence.

[0163] Preferably, the operation prediction module includes an operation type prediction unit and an operation parameter prediction unit; the operation type prediction unit is used to predict the type of the next operation, including click, input, scroll, these operation types are the basic units of Web page operation automation, by predicting these operation types, the model can build a complete operation sequence; the operation parameter prediction unit is used to predict the specific parameters of the operation, including the coordinates of the click and the text content of the input;

[0164] When the operation prediction module is working, the following steps are included:

[0165] S501: The operation prediction module receives the output from the Transformer encoder;

[0166] S502: extracting features related to operation prediction from the output of the Transformer encoder, and extracting features related to operation prediction, the features including historical information of the operation sequence, status information of the current page, and representation of user intention;

[0167] S503: using a multi-layer perceptron to classify the processed features, the model selects the most likely operation type from a predefined set of operation types to predict the type of the next operation;

[0168] S504: predicting the click coordinates for a click operation and predicting the input text content for an input operation through regression task and classification task methods;

[0169] S505: converting the predicted operation type and parameters into standardized operation instructions; the instructions are directly transmitted to the Web automation execution engine, and the execution engine executes the corresponding operation on the actual Web page;

[0170] S506: The execution result is used as feedback to update the state and training of the model; the model optimizes its prediction performance by continuously receiving feedback to better adapt to the dynamic Web environment.

[0171] Preferably, the Action Transformer model includes the following steps when working:

[0172] S1: Data preprocessing: Get the screenshot, text content and DOM structure of the current web page; preprocess these raw data, including image scaling, text segmentation, and DOM tree serialization;

[0173] S2: Feature extraction: The image processing submodule extracts visual features; the text processing submodule generates text embeddings; the DOM processing submodule generates structural features;

[0174] S3: Multimodal fusion: Use multi-head cross-attention mechanism to fuse features of different modalities;

[0175] S4: Transformer encoding: The fused features are processed through a multi-layer Transformer encoder to capture long-range dependencies and contextual information;

[0176] S5: Operation prediction: Based on the output of the encoder, predict the type and parameters of the next operation;

[0177] S6: Structured output: converting prediction results into standardized operational instructions;

[0178] S7: Execution and feedback: Pass the operation instructions to the execution engine; the execution results are used as feedback to update the status and training of the model.

[0179] Preferably, the training of the Action Transformer model is divided into two stages:

[0180] A1: Pre-training stage: Collect a large amount of diverse Web operation data; construct self-supervised learning tasks, such as operation sequence prediction and cross-modal matching; perform pre-training on large-scale data to learn common Web operation patterns;

[0181] A2: Fine-tuning stage: Collect relevant operational data for specific tasks or fields; use supervised learning and reinforcement learning methods for fine-tuning; optimize the performance of the model on specific tasks.

[0182] A multimodal Action Transformer model intelligent task execution method includes the following steps:

[0183] S801: Data collection and preprocessing: Get a screenshot of the current Web page; extract the text content of the page; get the DOM structure of the page; preprocess the collected data, including image scaling, text segmentation, and DOM tree serialization;

[0184] S802: Feature extraction: Use pre-trained convolutional neural networks to process page screenshots and extract visual features; use BERT and its variants to process page text content and generate text embeddings; use graph neural networks to process DOM tree structures and generate structured features;

[0185] S803: Multimodal feature fusion: The visual features, text embedding and structured features extracted in step S802 are input into the multi-head cross attention layer, and the features of different modalities are deeply fused through the multi-head cross attention mechanism; the fused multimodal feature representation is output;

[0186] S804: Transformer encoding: The fused features are input into a multi-layer Transformer encoder. Each layer of the Transformer encoder contains a self-attention mechanism and a feed-forward neural network. Through multi-layer processing, long-range dependencies are captured and the context of the operation sequence is understood.

[0187] S805: Operation prediction: Based on the output of the Transformer encoder, use a multi-layer perceptron to predict the operation; predict the type of the next operation, including click, input, and scroll; predict the specific parameters of the operation, including click coordinates and input text content;

[0188] S806: Structured output generation: convert the predicted operation type and parameters into a standardized instruction format, and generate a structured operation instruction, including the operation type, target element, and parameters;

[0189] S807: Instruction execution: The generated structured instructions are passed to the Web automation execution engine, and the execution engine performs corresponding operations on the actual Web page according to the instructions.

[0190] Preferably, when performing multimodal information encoding, the following steps are included:

[0191] S901: Obtain a screenshot of the current web page and convert it into an RGB pixel matrix format;

[0192] S902: extract all text content on the page, and perform word segmentation and cleaning to remove irrelevant characters and noise;

[0193] S903: Parse the DOM tree of the web page and serialize it into a node list and adjacency matrix format;

[0194] S904: Use the pre-trained convolutional neural network ResNet to process the page screenshot and extract the visual features of the image, including edges, textures, colors, and more advanced object and scene features;

[0195] S905: Convert the extracted features into vector representation;

[0196] S906: Use BERT and its variant RoBERTa to encode text content and generate text embedding;

[0197] S907: Convert text embedding into vector representation;

[0198] S908: Use graph neural network to process the DOM tree, capture the relationship and attribute information between DOM nodes to generate structured features;

[0199] S909: Input the feature vectors of the image, text, and DOM structure into the multi-head cross attention layer; through the attention mechanism, the model can automatically adjust the weights of different modal features to generate a fused multimodal feature representation;

[0200] S910: Integrate the fused multimodal feature representation into a unified vector, which contains all relevant information from the image, text and DOM structure;

[0201] S911: pass this unified vector to the subsequent Transformer encoder;

[0202] When performing structured vector processing, the following steps are included:

[0203] S1001: using a DOM parser to parse the HTML code of a web page into a DOM tree;

[0204] S1002: For each node in the DOM tree, extract its feature vector, the features including the node's tag name, attributes, and text content; the feature vector is generated by encoding each attribute of the node into a one-hot encoding or an embedding vector, and for the tag name, a predefined tag name embedding matrix is ​​used to map each tag name to a vector representation of a fixed length;

[0205] S1003: Use graph neural network to process the DOM tree to extract the relationship and structural information between nodes. GNN propagates information on the graph so that the representation of each node contains the information of its neighboring nodes. GNN uses the update rule of Graph Convolutional Network. The specific calculation formula is as follows:

[0206]

[0207] Among them, A is the adjacency matrix of the graph, D is the degree matrix, and H (l) is the node feature matrix of the lth layer, W (l) is the weight matrix of the lth layer, σ is the activation function;

[0208] S1004: Encode the operation on the web page into a vector representation of a fixed length by mapping the operation type to an embedding vector;

[0209] S1005: For each operation, extract its parameters and encode them, and the parameters are encoded into a numerical vector and an embedded vector;

[0210] S1006: fusing the feature vector of the Web page element and the feature vector of the operation by a concatenation and addition fusion method to generate a structured vector;

[0211] S1007: normalizing the fused feature vector by normalization and standardization methods to ensure the consistency of its dimension and value range;

[0212] S1008: Input the structured vector into the Transformer encoder and the operation prediction module.

[0213] As a preference, when converting the operation intention into behavior, the specific steps are as follows:

[0214] S1101: input natural language instructions;

[0215] S1102: performing natural language processing operations such as word segmentation, part-of-speech tagging, and named entity recognition on the instruction to extract key information and structure;

[0216] S1103: extracting semantic features of the instruction using the NLP model; inputting the features into the classifier to identify the user's operation intention;

[0217] S1104: extracting information of the target element from the instruction; if the operation requires additional parameters, parsing these parameters from the instruction;

[0218] S1105: extract visual features of the page screenshot using the image processing submodule; process the page text content using the text processing submodule to generate text embedding; process the DOM tree structure of the page using the DOM processing submodule to generate structured features; merge the above features with the semantic features of the instruction to obtain a more comprehensive context understanding;

[0219] S1106: Input the fused features into a multi-layer Transformer encoder to capture long-range dependencies and contextual information;

[0220] S1107: Based on the output of the encoder, use a multi-layer perceptron to predict the type and parameters of the next operation;

[0221] S1108: Convert the predicted operation type and parameters into a standardized instruction format, and optimize the instruction according to the context information;

[0222] S1109: passing the generated structured instruction to the Web automation execution engine to execute the corresponding operation, and the execution engine returns the operation result for updating the state and training of the model;

[0223] Among them, the semantic feature extraction formula in the intention recognition step is as follows:

[0224] Feature Vectors=NLP Model(Preprocessed Text);

[0225] Among them, the intent classification formula in the intent recognition step is as follows:

[0226] Intent = argmax i (w i Feature Vectors+b i );

[0227] Among them, in the parameter parsing step, target element recognition uses named entity recognition technology to extract target elements from instructions; parameter extraction uses regular expressions to parse the parameters required for the operation from the instructions;

[0228] Among them, the feature fusion formula in the multimodal information fusion step is as follows:

[0229] Fused Features=Fusion Layer(Image Features,Text Features,DOMFeatures,Semantic Features);

[0230] Among them, the Transformer encoding formula in the operation decision step is as follows:

[0231] Encoded Features=Transformer Encoder(Fused Features);

[0232] Among them, the operation prediction formula in the operation decision step is as follows:

[0233] Operation = argmax j (MLP(Encoded Features) j );

[0234] Among them, the structured output formula in the instruction generation step is as follows:

[0235] Structured Instruction=Format(Operation,Parameters);

[0236] The execution result output formula is as follows:

[0237] Result=Execution Engine (Structured Instruction).

[0238] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of those skilled in the art without departing from the spirit of the present invention.

Claims

1. A multimodal Action Transformer model; characterized by: Included are: A multimodal input processing module for simultaneously receiving and processing images, text, and structured data (DOM tree) as input; The multimodal fusion module is used to deeply fuse the features of different modalities using a multi-head cross-attention mechanism to establish cross-modal associations; Transformer encoder, which consists of a multi-layer Transformer encoder structure, each layer contains a self-attention mechanism and a feed-forward neural network to capture long-range dependencies and understand the context of the entire operation sequence; The operation prediction module is used to predict the type and parameters of the next operation based on the output of the Transformer encoder using a multi-layer perceptron, including but not limited to click, input, scroll and other operations and their specific parameters.

2. A multimodal Action Transformer model according to claim 1, characterized in that: The multimodal input processing module includes an image processing submodule, a text processing submodule and a DOM processing submodule; the image processing submodule is used to process the page screenshot using a pre-trained convolutional neural network to extract effective visual features; The text processing submodule is used to use BERT and its variant models to segment and embed the page text content to capture the text semantic information; the DOM processing submodule is used to use graph neural networks to process the DOM tree structure and generate structured feature representations to capture the relationship between page elements.

3. A multimodal Action Transformer model according to claim 2, characterized in that: The multimodal fusion module deeply fuses features from different modalities (image, text, DOM structure) to generate a comprehensive feature representation for subsequent operation prediction; The multimodal fusion module includes a feature input layer, a multi-head cross-attention mechanism unit and a fusion layer; the feature input layer receives feature vectors from the image processing submodule, the text processing submodule and the DOM processing submodule; the multi-head cross-attention mechanism unit allows the model to establish associations between different modalities and achieves deep fusion of features by calculating the attention weights between features of different modalities; the fusion layer further processes the output of the multi-head cross-attention mechanism to generate fusion features; When the multimodal fusion module is working, the following steps are included: S301: Obtain visual features of the page from the image processing submodule, obtain text embedding of the page from the text processing submodule, and obtain structural features of the page from the DOM processing submodule; S302: Input the extracted visual features, text embedding and structured features into the feature input layer of the multimodal fusion module; S303: Process the features of each modality separately, and calculate an attention weight matrix for each head to represent the degree of association between features of different modalities; S304: Perform weighted summation of features through the attention weight matrix to generate the output of each head; S305: splicing the outputs of all heads to obtain preliminary fusion features; S306: The fusion layer further processes the preliminary fusion features and generates the final fusion features through the fully connected layer and activation function; S307: Output the fused features to the Transformer encoder.

4. A multimodal Action Transformer model according to claim 3, characterized in that: The Transformer encoder includes an encoder input layer, a self-attention mechanism layer, a feedforward neural network layer, and an encoder output layer; the encoder input layer is used to receive the fused multimodal features as input; The self-attention mechanism layer is used to capture the long-range dependencies in the sequence by calculating the attention weights between each element and other elements, allowing the model to take into account the information of all other elements in the sequence when processing each input element; the feedforward neural network layer contains two linear transformations and an activation function, which are used to further process the output of the self-attention mechanism to extract higher-level features; the encoder output layer is used to generate encoded feature representations, which will be passed to the subsequent operation prediction module to predict the next operation type and parameters; The Transformer encoder includes the following steps when working: S401: Input the fused multimodal features into the Transformer encoder; S402: For each element in the input sequence, a query vector, a key vector, and a value vector are generated; S403: Calculate the dot product of the query vector and all key vectors, and use the softmax function to obtain the attention weight of each element to other elements; S404: Use the attention weights to perform weighted summation on the value vector to obtain the output of the self-attention mechanism; the output contains information about all elements in the sequence, and each element is weighted according to its relevance to the current element; S405: The output of the self-attention mechanism is passed to the feed-forward neural network, and the feed-forward neural network further extracts and transforms the output of the self-attention mechanism; S406: The output of the self-attention mechanism is processed by a feedforward neural network to generate an encoded feature representation including context information of the input sequence, and the feature representation is transmitted to the operation prediction module; S407: Repeat S401-S406, where each layer receives the output of the previous layer as input and generates a new encoded feature representation, so that the self-attention mechanism and feedforward neural network are usually stacked with multiple layers to more deeply capture the long-range dependencies and contextual information in the sequence.

5. A multimodal Action Transformer model according to claim 4, characterized in that: The operation prediction module includes an operation type prediction unit and an operation parameter prediction unit; the operation type prediction unit is used to predict the type of the next operation, including click, input, and scroll. These operation types are the basic units of Web page operation automation. By predicting these operation types, the model can build a complete operation sequence; the operation parameter prediction unit is used to predict the specific parameters of the operation, including the coordinates of the click and the text content of the input; When the operation prediction module is working, the following steps are included: S501: The operation prediction module receives the output from the Transformer encoder; S502: extracting features related to operation prediction from the output of the Transformer encoder, and extracting features related to operation prediction, the features including historical information of the operation sequence, status information of the current page, and representation of user intention; S503: using a multi-layer perceptron to classify the processed features, the model selects the most likely operation type from a predefined set of operation types to predict the type of the next operation; S504: predicting the click coordinates for a click operation and predicting the input text content for an input operation through regression task and classification task methods; S505: converting the predicted operation type and parameters into standardized operation instructions; The instructions are directly passed to the Web Automation Execution Engine, which performs the corresponding operations on the actual Web page; S506: The execution result is used as feedback to update the state and training of the model; The model continuously receives feedback to optimize its prediction performance and better adapt to the dynamic Web environment.

6. A multimodal Action Transformer model according to claim 5, characterized in that: The ActionTransformer model works by following these steps: S1: Data preprocessing: Get the screenshot, text content and DOM structure of the current web page; preprocess these raw data, including image scaling, text segmentation, and DOM tree serialization; S2: Feature extraction: The image processing submodule extracts visual features; The text processing submodule generates text embeddings; the DOM processing submodule generates structured features; S3: Multimodal fusion: Use multi-head cross-attention mechanism to fuse features of different modalities; S4: Transformer encoding: The fused features are processed through a multi-layer Transformer encoder to capture long-range dependencies and contextual information; S5: Operation prediction: Based on the output of the encoder, predict the type and parameters of the next operation; S6: Structured output: converting prediction results into standardized operational instructions; S7: Execution and feedback: Pass the operation instructions to the execution engine; the execution results are used as feedback to update the status and training of the model.

7. A multimodal Action Transformer model according to claim 6, characterized in that: The training of the ActionTransformer model is divided into two stages: A1: Pre-training stage: Collect a large amount of diverse Web operation data; construct self-supervised learning tasks, such as operation sequence prediction and cross-modal matching; perform pre-training on large-scale data to learn common Web operation patterns; A2: Fine-tuning stage: Collect relevant operational data for specific tasks or fields; use supervised learning and reinforcement learning methods for fine-tuning; optimize the performance of the model on specific tasks.

8. A multimodal Action Transformer model intelligent task execution method, characterized by: The following steps are included: S801: Data collection and preprocessing: Get a screenshot of the current Web page; extract the text content of the page; get the DOM structure of the page; preprocess the collected data, including image scaling, text segmentation, and DOM tree serialization; S802: Feature extraction: Use a pre-trained convolutional neural network to process the page screenshots and extract visual features; Use BERT and its variants to process page text content and generate text embeddings; use graph neural networks to process DOM tree structures and generate structured features; S803: Multimodal feature fusion: The visual features, text embedding and structured features extracted in step S802 are input into the multi-head cross attention layer, and the features of different modalities are deeply fused through the multi-head cross attention mechanism; the fused multimodal feature representation is output; S804: Transformer encoding: The fused features are input into a multi-layer Transformer encoder. Each layer of the Transformer encoder contains a self-attention mechanism and a feed-forward neural network. Through multi-layer processing, long-range dependencies are captured and the context of the operation sequence is understood. S805: Operation prediction: Based on the output of the Transformer encoder, use a multi-layer perceptron to predict the operation; predict the type of the next operation, including click, input, and scroll; predict the specific parameters of the operation, including click coordinates and input text content; S806: Structured output generation: convert the predicted operation type and parameters into a standardized instruction format, and generate a structured operation instruction, including the operation type, target element, and parameters; S807: Instruction execution: The generated structured instructions are passed to the Web automation execution engine, and the execution engine performs corresponding operations on the actual Web page according to the instructions.

9. The multimodal Action Transformer model intelligent task execution method according to claim 8, characterized in that: When encoding multimodal information, the following steps are included: S901: Obtain a screenshot of the current web page and convert it into an RGB pixel matrix format; S902: extract all text content on the page, and perform word segmentation and cleaning to remove irrelevant characters and noise; S903: Parse the DOM tree of the web page and serialize it into a node list and adjacency matrix format; S904: Use the pre-trained convolutional neural network ResNet to process the page screenshot and extract the visual features of the image, including edges, textures, colors, and more advanced object and scene features; S905: Convert the extracted features into vector representation; S906: Use BERT and its variant RoBERTa to encode text content and generate text embedding; S907: Convert text embedding into vector representation; S908: Use graph neural network to process the DOM tree, capture the relationship and attribute information between DOM nodes to generate structured features; S909: Input the feature vectors of the image, text, and DOM structure into the multi-head cross attention layer; through the attention mechanism, the model can automatically adjust the weights of different modal features to generate a fused multimodal feature representation; S910: Integrate the fused multimodal feature representation into a unified vector, which contains all relevant information from the image, text and DOM structure; S911: pass this unified vector to the subsequent Transformer encoder; When performing structured vector processing, the following steps are included: S1001: using a DOM parser to parse the HTML code of a web page into a DOM tree; S1002: For each node in the DOM tree, extract its feature vector, the features including the node's tag name, attributes, and text content; the feature vector is generated by encoding each attribute of the node into a one-hot encoding or an embedding vector, and for the tag name, a predefined tag name embedding matrix is ​​used to map each tag name to a vector representation of a fixed length; S1003: Use graph neural network to process the DOM tree to extract the relationship and structural information between nodes. GNN propagates information on the graph so that the representation of each node contains the information of its neighboring nodes. GNN uses the update rule of Graph Convolutional Network. The specific calculation formula is as follows: Where A is the adjacency matrix of the graph, D is the degree matrix, and H (l) is the node feature matrix of the lth layer, W (l) is the weight matrix of the lth layer, σ is the activation function; S1004: Encode the operation on the web page into a vector representation of a fixed length by mapping the operation type to an embedding vector; S1005: For each operation, extract its parameters and encode them, and the parameters are encoded into a numerical vector and an embedded vector; S1006: fusing the feature vector of the Web page element and the feature vector of the operation by a concatenation and addition fusion method to generate a structured vector; S1007: normalizing the fused feature vector by normalization and standardization methods to ensure the consistency of its dimension and value range; S1008: Input the structured vector into the Transformer encoder and the operation prediction module.

10. The multimodal Action Transformer model intelligent task execution method according to claim 9, characterized in that: When converting operation intention into behavior, the specific steps are as follows: S1101: input natural language instructions; S1102: performing natural language processing operations such as word segmentation, part-of-speech tagging, and named entity recognition on the instruction to extract key information and structure; S1103: extracting semantic features of the instruction using the NLP model; inputting the features into the classifier to identify the user's operation intention; S1104: extracting information of the target element from the instruction; If the operation requires additional parameters, these are parsed from the instruction; S1105: extracting visual features of the page screenshot using an image processing submodule; Use the text processing submodule to process the page text content and generate text embedding; use the DOM processing submodule to process the DOM tree structure of the page and generate structured features; Combine the above features with the semantic features of the instruction to get a more comprehensive contextual understanding; S1106: Input the fused features into a multi-layer Transformer encoder to capture long-range dependencies and contextual information; S1107: Based on the output of the encoder, use a multi-layer perceptron to predict the type and parameters of the next operation; S1108: Convert the predicted operation type and parameters into a standardized instruction format, and optimize the instruction according to the context information; S1109: passing the generated structured instruction to the Web automation execution engine to execute the corresponding operation, and the execution engine returns the operation result for updating the state and training of the model; Among them, the semantic feature extraction formula in the intention recognition step is as follows: FeatureVectors=NLPModel(PreprocessedText); Among them, the intent classification formula in the intent recognition step is as follows: Intent=argmax i (w i ·FeatureVectors+b i ); Among them, in the parameter parsing step, target element recognition uses named entity recognition technology to extract target elements from instructions; parameter extraction uses regular expressions to parse the parameters required for the operation from the instructions; Among them, the feature fusion formula in the multimodal information fusion step is as follows: FusedFeatures=Fusion Layer(ImageFeatures,Text Features,DOMFeatures,SemanticFeatures); Among them, the Transformer encoding formula in the operation decision step is as follows: EncodedFeatures=Transformer Encoder(FusedFeatures); Among them, the operation prediction formula in the operation decision step is as follows: Operation=argmax j (MLP(EncodedFeatures) j ); Among them, the structured output formula in the instruction generation step is as follows: StructuredInstruction=Format(Operation,Parameters); The execution result output formula is as follows: Result=Execution Engine(StructuredInstruction).

Images