A Deep Learning-Based Intelligent Automated Execution System for Web Business Processes

Abstract

This invention discloses a deep learning-based intelligent automated execution system for web business processes, belonging to the field of web automation technology. It includes a data acquisition and semantic parsing module for acquiring business process descriptions and multimodal data of web pages, generating a structured business process representation; a deep learning feature extraction module for performing deep learning feature extraction on the multimodal data to obtain corresponding structural feature vectors, visual feature vectors, and text feature vectors; a cross-modal association and fusion module for generating a unified page feature representation; a page element intelligent recognition module for identifying and locating target elements based on the unified feature representation; an instruction generation module for generating operation instruction sequences; and an execution control module for executing instructions and dynamically adjusting strategies. This invention establishes a complete automated chain from business description to technical execution, supports conditional branching and complex dependency management, and reduces manual programming and maintenance workload.

Description

Technical Field

[0001] This invention relates to the field of Web automation technology, and in particular to an intelligent automated execution system for Web business processes based on deep learning. Background Technology

[0002] As enterprises deepen their digital transformation, the demand for automation of web business processes is growing. Enterprise applications contain a large number of repetitive web operations, such as data entry, form filling, and information retrieval. These tasks are characterized by high regularity and large volume, and urgently require intelligent processing through automation technology.

[0003] Existing web automation technologies mainly fall into three categories: recording and playback, DOM parsing, and computer vision. Recording and playback methods are sensitive to changes in page structure and have high maintenance costs; DOM parsing methods rely on fixed element positioning strategies and have low accuracy when dealing with dynamic content and complex styles; computer vision methods lack a deep understanding of page semantics and are prone to misidentification in complex interfaces. Current technologies suffer from insufficient intelligence, poor adaptability to complex pages in modern web applications, and a lack of comprehensive utilization of multimodal information, limiting the stability and accuracy of automated execution. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a deep learning-based intelligent automated execution system for web business processes, which addresses the technical problems of insufficient intelligence and poor adaptability to complex pages in existing web business process automation technologies.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a deep learning-based intelligent automated execution system for web business processes, comprising:

[0008] The data acquisition and semantic parsing module is used to acquire multimodal data from business process descriptions and web pages, and to perform semantic parsing on the business process descriptions to generate a structured business process representation.

[0009] The deep learning feature extraction module is used to perform deep learning feature extraction on multimodal data to obtain corresponding structural feature vectors, visual feature vectors, and text feature vectors.

[0010] The cross-modal association and fusion module is used to fuse structural feature vectors, visual feature vectors, and text feature vectors through a cross-modal association mechanism to generate a unified page feature representation;

[0011] The page element intelligent recognition module is used to build a page element intelligent recognition model based on a unified page feature representation to identify and locate target elements;

[0012] The instruction generation module is used to generate a sequence of operation instructions based on the structured business process representation and target element recognition results, using an instruction generation algorithm.

[0013] The execution control module is used to execute the sequence of operation instructions and dynamically adjust the element recognition strategy and operation strategy based on the execution feedback.

[0014] As a preferred embodiment of the deep learning-based intelligent automated execution system for web business processes according to the present invention, the process of generating a structured business process representation through semantic parsing of the business process description includes:

[0015] The business process description is preprocessed to extract key business semantic fragments and initial operation parameters;

[0016] Key business semantic fragments are input into a multi-layer semantic parsing model to generate hierarchical semantic representations;

[0017] Based on hierarchical semantic representation, entity recognition and relationship extraction are performed simultaneously through a joint extraction framework to obtain candidate business entities and relationships between entities.

[0018] Candidate business entities are linked with pre-built business knowledge graphs and disambiguated to generate standardized entity representations;

[0019] Based on standardized entity representations, inter-entity relationships, and initial operation parameters, a structured business process representation is generated through a process construction algorithm.

[0020] As a preferred embodiment of the deep learning-based intelligent automated execution system for web business processes according to the present invention, the generation of a structured business process representation through a process construction algorithm includes:

[0021] Based on normalized entity representation and relationships between entities, a business process diagram is created using a graph construction algorithm;

[0022] The initial operation parameters are matched and bound to the corresponding operation nodes through a parameter binding algorithm;

[0023] Based on the conditional relationships between entities, a conditional judgment logic is extracted through a conditional expression parser to generate a set of conditional rules.

[0024] Based on business process diagrams and sets of conditional rules, a structured business process representation is generated through a process integration algorithm.

[0025] As a preferred embodiment of the intelligent automated execution system for web business processes based on deep learning according to the present invention, the multimodal data includes DOM structure data, page image data, and page text content data; the processing flow of the deep learning feature extraction module includes:

[0026] A graph neural network is used to encode the DOM structure data, extract node attribute features, hierarchical relationship features, semantic role features and interaction state features, and generate a structure feature vector.

[0027] A visual coding model based on an attention mechanism is used to process page image data, extract spatial layout features, visual appearance features, and element boundary features, and generate visual feature vectors.

[0028] Semantic encoding is performed on the page text content data to extract text semantic features and contextual features, generating text feature vectors.

[0029] As a preferred embodiment of the deep learning-based intelligent automated execution system for web business processes of this invention, the processing flow of the cross-modal association and fusion module includes:

[0030] The structural feature vectors, visual feature vectors, and text feature vectors are dimensionally aligned and feature standardized to map them to a unified feature space.

[0031] Intramodal association weights and cross-modal association weights are calculated separately using a multimodal attention mechanism;

[0032] Based on the intramodal correlation weights, the feature vectors of each modality are weighted and aggregated to generate optimized feature representations for each modality;

[0033] Based on cross-modal association weights, intermodal feature alignment and fusion are performed on the optimized feature representations of each modality. The contribution of each optimized feature representation is dynamically adjusted through a gating mechanism to generate a preliminary fused feature representation.

[0034] The preliminary fused feature representation is processed by residual connection to generate a unified page feature representation.

[0035] As a preferred embodiment of the deep learning-based intelligent automated execution system for web business processes of this invention, the processing flow of the page element intelligent recognition module includes:

[0036] Based on a unified page feature representation, an object detection network is used to identify interactive elements on the page and generate a set of candidate elements containing bounding box coordinates and detection confidence scores.

[0037] The element classification module performs type identification and attribute parsing on the candidate element set to obtain the element type and operation attributes of each candidate element.

[0038] Spatial matching is performed between the bounding box coordinates of candidate elements and the DOM structure to generate a corresponding positioning identifier for each candidate element;

[0039] Based on detection confidence, element type, and operation attributes, candidate elements are filtered and sorted to determine the target element;

[0040] Perform state detection on the target element, obtain the interaction state of the target element, and generate the target element recognition result.

[0041] As a preferred embodiment of the deep learning-based intelligent automated execution system for web business processes of this invention, the processing flow of the instruction generation module includes:

[0042] Based on the target objects of the operation nodes in the structured business process representation, a matching and mapping is performed with the target element recognition results;

[0043] Based on the operation type in the operation node sequence and the element type of the target element, a parameterized instruction template is generated by matching through the instruction template library.

[0044] Bind the operation parameters in the operation node sequence and the location identifier in the target element recognition result to the parameterized instruction template to generate an executable operation instruction;

[0045] Based on conditional branching rules, a conditional control structure is created using a conditional instruction generator;

[0046] Based on the dependencies between nodes, the executable operation instructions and condition control structures are sorted and optimized to generate an operation instruction sequence.

[0047] Secondly, this invention provides a deep learning-based intelligent automated execution method for web business processes, comprising:

[0048] Acquire multimodal data from business process descriptions and web pages, and perform semantic parsing on the business process descriptions to generate a structured business process representation;

[0049] Deep learning is used to extract features from multimodal data to obtain corresponding structural feature vectors, visual feature vectors, and text feature vectors.

[0050] By fusing structural feature vectors, visual feature vectors, and text feature vectors through a cross-modal association mechanism, a unified page feature representation is generated.

[0051] A page element intelligent recognition model is built based on a unified page feature representation to identify and locate target elements;

[0052] Based on the structured business process representation and target element recognition results, an operation instruction sequence is generated through an instruction generation algorithm;

[0053] Execute the sequence of operation instructions and dynamically adjust the element recognition strategy and operation strategy based on the execution feedback.

[0054] Thirdly, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the computer program, when executed by the processor, implements any step of the deep learning-based intelligent automated execution system for web business processes as described in the first aspect of the present invention.

[0055] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the deep learning-based intelligent automated execution system for web business processes as described in the first aspect of the present invention.

[0056] The beneficial effects of this invention are as follows: This invention achieves automatic conversion from natural language business descriptions to structured process representations through a data acquisition and semantic parsing module, significantly reducing the technical threshold for business process automation; it extracts deep features from DOM structure, page images, and text content through a deep learning feature extraction module, obtaining richer and more accurate page representation information; it constructs a unified feature representation through a cross-modal association and fusion module, effectively integrating the complementary advantages of multiple modal information; and it achieves high-precision target element recognition and positioning through a page element intelligent recognition module, improving the system's adaptability to dynamic content, complex styles, and diverse UI frameworks. Attached Figure Description

[0057] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is the overall architecture diagram of a deep learning-based intelligent automated execution system for web business processes.

[0059] Figure 2 This is a flowchart illustrating the semantic parsing of business processes for a deep learning-based intelligent automated execution system for web business processes.

[0060] Figure 3 This is a flowchart of the cross-modal association and fusion module of a deep learning-based intelligent automated execution system for web business processes.

[0061] Figure 4 This is a flowchart of a deep learning-based intelligent automated execution method for web business processes. Detailed Implementation

[0062] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0063] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0064] Secondly, the term "an embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single embodiment or an embodiment selectively excluded from other embodiments.

[0065] Reference Figures 1-4 This is one embodiment of the present invention, which provides a deep learning-based intelligent automated execution system for web business processes. The overall architecture diagram is shown below. Figure 1 As shown, the system includes the following steps:

[0066] The data acquisition and semantic parsing module is used to acquire multimodal data from business process descriptions and web pages, and to perform semantic parsing on the business process descriptions to generate a structured business process representation.

[0067] The deep learning feature extraction module is used to perform deep learning feature extraction on multimodal data to obtain corresponding structural feature vectors, visual feature vectors, and text feature vectors.

[0068] The cross-modal association and fusion module is used to fuse structural feature vectors, visual feature vectors, and text feature vectors through a cross-modal association mechanism to generate a unified page feature representation.

[0069] The page element intelligent recognition module is used to build a page element intelligent recognition model based on a unified page feature representation to identify and locate target elements.

[0070] The instruction generation module is used to generate a sequence of operation instructions based on the structured business process representation and the target element recognition results, using an instruction generation algorithm.

[0071] The execution control module is used to execute the sequence of operation instructions and dynamically adjust the element recognition strategy and operation strategy based on the execution feedback.

[0072] Specifically, business process descriptions can be obtained in various ways, including but not limited to: direct input through the user interface, data import from business systems, extraction through document parsing, and generation through operation recording. Multimodal data refers to data of different types and sources in a web page, including DOM structure data, page image data, and page text content data.

[0073] The multimodal data acquisition process is as follows: The target web page is accessed through a web browser engine to obtain complete DOM structure data, including HTML node hierarchy, node attribute information, CSS style properties, and JavaScript event binding information; a screenshot of the target web page is taken to obtain page image data, including page layout information, visual element positions, visual features, and interface components such as icons and buttons; all text node content is extracted from the DOM structure data to obtain page text content data, including tag text, placeholder text, button text, and tooltip text; finally, the above three types of data are preprocessed and standardized, including data cleaning, coordinate normalization, and encoding format unification, to generate a unified format multimodal data set.

[0074] Furthermore, the business process description undergoes text preprocessing to extract key business semantic fragments and initial operation parameters. Text preprocessing includes text cleaning and semantic segmentation, while initial operation parameters include input data, selection options, and configuration parameters. The key business semantic fragments are then input into a multi-layer semantic parsing model, which generates hierarchical semantic representations. These hierarchical representations include word-level and sentence-level semantics. The multi-layer semantic parsing model is built on an attention mechanism and achieves a progressive understanding from local lexical semantics to global sentence semantics through multi-layer encoding.

[0075] Furthermore, based on hierarchical semantic representation, entity recognition and relation extraction are performed simultaneously through a joint extraction framework to obtain candidate business entities and relationships between entities. The steps are as follows: The hierarchical semantic representation is input into a shared encoder based on a Transformer architecture and a multi-head attention mechanism to generate a unified context representation vector; based on the context representation vector, entity boundary recognition and relation type prediction are performed simultaneously through a joint decoding mechanism; the entity recognition results and relation extraction results are collaboratively optimized through a consistency constraint mechanism to generate the final business entities and relationships between entities. Candidate business entities include operation objects, action types, and constraints, while relationships between entities include sequential relationships, dependency relationships, and conditional relationships.

[0076] Furthermore, candidate business entities are linked and disambiguated with a pre-built business knowledge graph to generate standardized entity representations. The steps are as follows: Based on the textual features of candidate business entities, similar candidate matching entities are retrieved from the business knowledge graph; the retrieved candidate matching entities are scored and ranked by semantic similarity calculation to generate a matching score ranking list; for cases with multiple highly similar candidate entities, disambiguation is performed based on contextual semantics and domain constraint rules to determine a unique matching entity; quality assessment is performed based on the confidence of the matching results, and high-confidence matching results are converted into standardized entity representations in the knowledge graph; successfully linked entities are output as standardized entity representations, while unsuccessfully linked entities retain their original form and are marked, ultimately generating a structured representation containing standardized business entities.

[0077] It should be noted that the pre-built business knowledge graph includes standardized web business operation entities, UI element types, and interaction action types, and is built offline through a combination of business process data mining and domain expert annotation.

[0078] Furthermore, based on the normalized entity representation, inter-entity relationships, and initial operation parameters, a structured business process representation is generated through a process construction algorithm. The semantic parsing flowchart of the business process is as follows: Figure 2 As shown, the process includes: creating a business process diagram based on standardized entity representations and inter-entity relationships using a graph construction algorithm; the business process diagram contains operation nodes and connecting edges; operation nodes contain operation types and target objects, and connecting edges contain relationship types and timing information. An initial operation parameters are matched and bound to the corresponding operation nodes using a parameter binding algorithm. Based on the conditional relationships in the inter-entity relationships, a conditional expression parser extracts the conditional judgment logic, generating a set of conditional rules; the set of conditional rules contains judgment conditions, execution branches, and triggering nodes.

[0079] Furthermore, based on the business process diagram and the set of conditional rules, a structured business process representation is generated through a process integration algorithm. The steps are as follows: Based on the operation nodes and connecting edges of the business process diagram, an operation node sequence is generated through topological sorting, maintaining the temporal relationship and execution order between nodes; the set of conditional rules is matched with the operation node sequence, and conditional branch rules are inserted at the corresponding positions; the conditional branch rules include judgment conditions and execution branches; based on the connection edge relationship types of the business process diagram, the dependencies between nodes are extracted and labeled, including sequence dependencies, data dependencies, and control dependencies; the operation node sequence, conditional branch rules, and dependencies between nodes are encapsulated to generate a structured business process representation, where the operation node sequence includes operation type, target object, and operation parameters.

[0080] Preferably, this invention achieves accurate conversion of business process descriptions to structured representations by constructing a multi-layer semantic parsing model and a joint extraction framework. Compared to existing rule-based or simple NLP techniques, this invention employs a BERT-based joint extraction framework to simultaneously perform entity recognition and relation extraction, avoiding the error propagation problem of traditional pipeline methods and improving the recognition accuracy of business entities and relations. Through entity linking and disambiguation processing of the business knowledge graph, the ambiguity and non-standardization of business terms are resolved, resulting in more accurate and standardized structured business process representations.

[0081] In one embodiment, the deep learning feature extraction module includes three parallel processing flows. First, a graph neural network is used to encode the DOM structure data, extracting node attribute features, hierarchical relationship features, semantic role features, and interaction state features to generate a structural feature vector.

[0082] Specifically, a DOM structure graph is constructed based on the hierarchical relationship of HTML nodes in the DOM structure data. Attribute encoding is performed on the nodes in the DOM structure graph to extract node tag types, CSS style properties, and position coordinates, generating node attribute features. Through the message passing mechanism of a graph neural network, the DOM structure graph is aggregated in multiple layers to extract hierarchical relationship features and semantic role features. Hierarchical relationship features reflect the node's hierarchy information in the DOM tree, while semantic role features are determined based on node tag types and style properties. The interaction state of nodes is encoded based on JavaScript event binding information to generate interaction state features. Finally, the node attribute features, hierarchical relationship features, semantic role features, and interaction state features are concatenated to generate a structural feature vector.

[0083] Secondly, a visual encoding model based on an attention mechanism is used to process the page image data, extracting spatial layout features, visual appearance features, and element boundary features to generate a visual feature vector. Specifically, a convolutional neural network is used to perform multi-scale feature extraction on the page image data to obtain a low-level visual feature map, which contains edge, texture, and color information. Based on the low-level visual feature map, spatial layout features are extracted using a spatial attention mechanism. The spatial attention mechanism is used to identify key areas and element distributions on the page. An object detection algorithm is used to identify interface elements on the page, extracting element boundary features and visual appearance features. Interface elements include buttons, input boxes, icons, and text areas. The spatial layout features, visual appearance features, and element boundary features are concatenated to generate a visual feature vector.

[0084] Finally, a pre-trained language model is used to semantically encode the page text content data, extract text semantic features and contextual features, and generate text feature vectors. The pre-trained language model includes, but is not limited to, BERT and RoBERTa.

[0085] In the feature extraction process, this invention is specifically designed to meet the unique needs of automated execution of web business processes: In DOM structure feature extraction, semantic role features are used to identify the business function type of page elements, and interaction state features are used to determine the operability of elements, ensuring that the automated system can accurately identify and locate executable operation targets. In visual feature extraction, element boundary features are used to achieve precise spatial positioning of key operation elements such as buttons and input boxes. In text feature extraction, contextual association features are used to understand the semantic correspondence between tags and input fields. This targeted multimodal feature extraction design effectively solves the problems of inaccurate identification, imprecise positioning, and inadequate semantic understanding of operation elements in traditional methods, improving the success rate and stability of automated execution of web business processes.

[0086] In one embodiment, the flowchart of the cross-modal correlation fusion module is as follows: Figure 3 As shown, the process includes: dimensional alignment and feature normalization of structural feature vectors, visual feature vectors, and text feature vectors, mapping them to a unified feature space. Intra-modal association weights and cross-modal association weights are calculated using a multimodal attention mechanism. Intra-modal association weights measure the importance of features within each modality and are calculated using a self-attention mechanism; cross-modal association weights measure the semantic relevance between different modalities and are calculated using a cross-attention mechanism. Based on the intra-modal association weights, the feature vectors of each modality are weighted and aggregated to generate optimized feature representations for each modality.

[0087] Preferably, based on cross-modal correlation weights, inter-modal feature alignment and fusion are performed on the optimized feature representations of each modality. The contribution of each optimized feature representation is dynamically adjusted through a gating mechanism to generate a preliminary fused feature representation. The steps are as follows: Based on cross-modal correlation weights, each optimized feature representation of each modality is mapped to a unified feature dimension through a linear projection layer to achieve inter-modal feature alignment; dynamic contribution weights are calculated for each optimized feature representation of each modality through a gating network; the gating network adopts a multilayer perceptron structure and outputs the weight coefficients of each modality; the dynamic contribution weights are normalized to ensure that the sum of the weights of each modality is 1; the aligned feature representations of each modality are weighted and fused according to the normalized weights to generate a preliminary fused feature representation. The weighted fusion achieves cross-modal information integration through a weighted summation method.

[0088] Furthermore, residual connection processing is applied to the preliminary fused feature representation to generate a unified page feature representation. The residual connection processing aims to maintain the integrity of DOM structure information, visual appearance information, and textual semantic information during multimodal feature fusion, preventing important features of one modality from being obscured by features of other modalities during cross-modal fusion, and ensuring that the final unified page feature representation can comprehensively reflect the multidimensional features of page elements.

[0089] Preferably, the cross-modal association and fusion module adopts a hierarchical and progressive multimodal fusion architecture. Through intra-modal self-optimization and cross-modal dynamic balancing mechanisms, it solves the two major technical challenges of "information imbalance" and "feature loss" in multimodal information fusion of web pages. The synergistic effect brought about by this hierarchical and progressive architecture is that, while ensuring the integrity of features of each modality, it achieves true complementary fusion. This allows the system to utilize the precision of the DOM structure and the intuitiveness of visual information, while also combining the richness of textual semantics, avoiding the technical defect of strong modalities suppressing weak modalities in simple fusion methods.

[0090] In one embodiment, the processing flow of the page element intelligent recognition module includes: based on a unified page feature representation, identifying interactive elements in the page through an object detection network, and generating a candidate element set containing bounding box coordinates and detection confidence scores. The steps are as follows: performing feature preprocessing on the unified page feature representation, including dimensional adjustment and data format conversion, to adapt it to the input requirements of the object detection network; inputting the preprocessed feature representation into the object detection network for forward inference, generating candidate region proposals through convolutional feature extraction and multi-scale detection heads; performing non-maximum suppression processing on the candidate region proposals to remove redundant detection boxes with high overlap, with non-maximum suppression filtering based on detection confidence scores and bounding box overlap rates; and performing post-processing on the filtered detection results to extract bounding box coordinates and corresponding detection confidence scores to generate a candidate element set, where each element in the candidate element set contains the coordinates of the top-left and bottom-right corners and a confidence score.

[0091] Furthermore, the candidate element set is subjected to type identification and attribute parsing through the element classification module to obtain the element type and operation attributes of each candidate element. The steps are as follows: Based on the bounding box coordinates of the candidate elements, the corresponding local feature regions are extracted from the unified page feature representation. The local feature regions contain the structural features, visual features, and text features of the candidate elements. The local feature regions are input into the element type classifier for type identification, and the type labels of the candidate elements are output. The type labels include Web interface element types such as buttons, input boxes, links, and drop-down menus. The local feature regions are input into the attribute parsing network to extract the operation attribute information of the candidate elements. The operation attribute information includes interactive capability identifiers such as clickability, input capability, and selectability. The type labels and operation attribute information are integrated to generate a complete element description of each candidate element. The complete element description contains a structured representation of the element type and operation attributes.

[0092] Furthermore, spatial matching is performed between the bounding box coordinates of candidate elements and the DOM structure. A coordinate mapping algorithm is used to generate a corresponding positioning identifier for each candidate element, which performs geometric matching based on the pixel coordinates of the element and the layout position of the DOM node. Based on detection confidence, element type, and operation attributes, candidate elements are filtered and sorted to determine the target element. The filtering and sorting includes confidence threshold filtering, type matching verification, and attribute compatibility checking. Confidence threshold filtering is used to filter low-quality detection results, type matching verification is used to filter element types that meet the operation requirements, and attribute compatibility checking is used to verify whether the element supports the target operation.

[0093] Furthermore, state detection is performed on the target element to obtain its interaction state and generate a target element recognition result. The target element recognition result includes bounding box coordinates, element type, interaction state, and location identifier. The interaction state includes the target element's availability state, visibility state, and content state.

[0094] Ideally, this intelligent element recognition module significantly improves the accuracy and efficiency of web interface element recognition, effectively addressing the technical challenges of dynamic content, complex layouts, and asynchronous loading in modern web applications. This module solves the key technical problem of traditional DOM parsing-based methods failing in complex page scenarios, achieving precise positioning and comprehensive attribute acquisition of various interactive elements. This technical solution has good generalization capabilities, adapting to different web frameworks and UI design styles, providing a reliable foundation for target recognition and state awareness for web automation operations, and improving the intelligence level and success rate of automated operations.

[0095] In one embodiment, the processing flow of the instruction generation module includes: matching and mapping the target object of the operation node in the structured business process representation with the target element identification result to establish the correspondence between business operations and specific page elements. Based on the operation type in the operation node sequence and the element type of the target element, a parameterized instruction template is generated by matching through an instruction template library. The instruction template library contains standard operation instruction templates corresponding to different element types. The steps are as follows: filtering a set of candidate templates from the instruction template library according to the operation type; determining compatible instruction templates based on the target element type; adjusting the instruction template according to the element operation attributes through a template adaptation algorithm; and mapping the operation parameters to template placeholders to generate the parameterized instruction template.

[0096] Furthermore, the operation parameters in the operation node sequence and the location identifiers in the target element recognition results are bound to the parameterized instruction template to generate executable operation instructions. Based on the conditional branching rules, a conditional control structure is created through a conditional instruction generator, with the following steps: parsing the conditional expressions in the conditional branching rules to extract conditional variables, comparison operators, and threshold parameters; generating corresponding state detection instructions based on the type and source of the conditional variables; constructing a conditional judgment logic module based on the comparison operators and threshold parameters, which performs Boolean operations on the state detection results and outputs the judgment results; generating corresponding branch execution sequences for different branch paths in the conditional branching rules, each containing a set of operation instructions to be executed when the branch conditions are met; and structuring the state detection instructions, conditional judgment logic, and branch execution sequences to form a complete conditional control structure, which supports basic condition combinations and second-level nested logic processing.

[0097] Furthermore, based on the dependencies between nodes, the executable operation instructions and conditional control structures are sorted and optimized to generate an operation instruction sequence. The steps are as follows: Parse the preconditions, postconditions, and data dependencies in the structured business process representation to establish dependency mappings between operation nodes; traverse the dependency graph using a topological sorting algorithm to ensure that preconditions are executed before postconditions; for conditional control structures, insert them as special nodes into appropriate positions in the sorted sequence to ensure that condition judgments are executed after dependent operations are completed; sort the execution priority based on operation type and element complexity to avoid concurrent operation conflicts for the same target element, generating an ordered operation instruction sequence.

[0098] Preferably, the operation instruction generation module automatically converts abstract business processes into concrete executable instructions, establishing a correspondence between business logic and page operations. Through a mapping mechanism between business objects and page elements, combined with a templated instruction generation method, high-level business descriptions can be converted into standardized page operation instructions. This module supports conditional branch control and dependency handling, enabling it to handle logical branches and execution order constraints in business scenarios. Through parameterized templates and sorting optimization mechanisms, the generated operation instruction sequences have a unified format specification and a clear execution order, providing an instruction foundation for the automated execution of web business processes. This allows business personnel to build automated processes based on business descriptions without directly writing operation instructions.

[0099] It should be noted that the operation command execution and dynamic adjustment mechanism in this technical solution achieves closed-loop optimization based on real-time feedback processing. Execution feedback includes multi-dimensional information such as operation success status, element response status, page change information, and error codes, which are structured through preset feedback parsing rules. The dynamic adjustment strategy employs different optimization methods based on the feedback type: when element recognition fails, the detection threshold and feature weight parameters are adjusted; when operation execution fails, the command execution strategy and retry parameters are modified; when the page state is abnormal, the element recognition process is retried. Strategy adjustments use incremental optimization to avoid instability caused by large parameter changes, and an upper limit on the number of adjustments and an effect evaluation mechanism are set to ensure convergence. This feedback optimization mechanism improves the adaptability of the automated process to dynamic page changes and abnormal situations, ensuring the stability and reliability of business process execution.

[0100] This embodiment also provides a method for intelligent automated execution of web business processes based on deep learning, the flowchart of which is shown below. Figure 4 As shown, the method includes: acquiring multimodal data of business process description and web page; performing semantic parsing on the business process description to generate a structured business process representation; performing deep learning feature extraction on the multimodal data to obtain corresponding structural feature vectors, visual feature vectors, and text feature vectors; fusing the structural feature vectors, visual feature vectors, and text feature vectors through a cross-modal association mechanism to generate a unified page feature representation; constructing an intelligent page element recognition model based on the unified page feature representation to identify and locate target elements; generating an operation instruction sequence through an instruction generation algorithm based on the structured business process representation and the target element recognition results; executing the operation instruction sequence and dynamically adjusting the element recognition strategy and operation strategy based on the execution feedback.

[0101] This embodiment also provides a computer device applicable to a deep learning-based intelligent automated execution system for web business processes, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the deep learning-based intelligent automated execution system for web business processes proposed in the above embodiment.

[0102] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0103] This embodiment also provides a storage medium storing a computer program. When executed by a processor, the program implements the intelligent automated execution system for web business processes based on deep learning, as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0104] In summary, this invention achieves automatic conversion from natural language business descriptions to structured process representations through a data acquisition and semantic parsing module, significantly reducing the technical threshold for business process automation; it extracts deep features from DOM structure, page images, and text content through a deep learning feature extraction module, obtaining richer and more accurate page representation information; it constructs a unified feature representation through a cross-modal association and fusion module, effectively integrating the complementary advantages of multiple modal information; and it achieves high-precision target element recognition and positioning through a page element intelligent recognition module, improving the system's adaptability to dynamic content, complex styles, and diverse UI frameworks.

[0105] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A deep learning based intelligent automation execution system for web business process, characterized in that: The method comprises the following steps: a data acquisition and semantic analysis module is used to acquire multi-modal data of a business process description and a web page, and to perform semantic analysis on the business process description to generate a structured business process representation; a deep learning feature extraction module is used to perform deep learning feature extraction on the multi-modal data to obtain corresponding structural feature vectors, visual feature vectors and text feature vectors; a cross-modal association and fusion module is used to fuse the structural feature vectors, visual feature vectors and text feature vectors through a cross-modal association mechanism to generate a unified page feature representation; a page element intelligent identification module is used to construct a page element intelligent identification model based on the unified page feature representation, and to identify and locate a target element; an instruction generation module is used to generate an operation instruction sequence through an instruction generation algorithm based on the structured business process representation and the target element identification result; an execution control module is used to execute the operation instruction sequence, and to dynamically adjust an element identification strategy and an operation strategy according to execution feedback; the processing flow of the page element intelligent identification module comprises the following steps: based on the unified page feature representation, a target detection network is used to identify interactive elements in a page to generate a candidate element set containing boundary box coordinates and detection confidence; an element classification module is used to identify the types and attributes of the candidate elements in the candidate element set to obtain the element types and operation attributes of the candidate elements; based on the boundary box coordinates of the candidate elements and a DOM structure, spatial matching is performed to generate a corresponding positioning identifier for each candidate element; based on the detection confidence, element types and operation attributes, the candidate elements are filtered and sorted to determine a target element; state detection is performed on the target element to obtain the interactive state of the target element, and a target element identification result is generated; the processing flow of the instruction generation module comprises the following steps: based on the target object of an operation node in the structured business process representation, the target element identification result is matched and mapped; based on the operation types in the operation node sequence and the element types of the target elements, a parameterized instruction template is generated through matching of an instruction template library; operation parameters in the operation node sequence and the positioning identifier in the target element identification result are bound to the parameterized instruction template to generate executable operation instructions; based on conditional branching rules, a conditional instruction generator is used to create a conditional control structure; based on the inter-node dependency relationship, the executable operation instructions and the conditional control structure are sorted and optimized to generate an operation instruction sequence. 2.The deep learning based web business process intelligent automation execution system of claim 1, wherein: the semantic analysis on the business process description to generate a structured business process representation comprises the following steps: text preprocessing is performed on the business process description to extract key business semantic segments and initial operation parameters; the key business semantic segments are input into a multi-layer semantic analysis model to generate hierarchical semantic representations; based on the hierarchical semantic representations, entity recognition and relationship extraction are simultaneously performed through a joint extraction framework to obtain candidate business entities and inter-entity relationships; entity linking and disambiguation processing are performed on the candidate business entities and a pre-constructed business knowledge graph to generate standardized entity representations; Based on the normalized entity representation, inter-entity relationship and initial operation parameter, a structured business process representation is generated by a process construction algorithm. 3.The deep learning based web business process intelligent automation execution system of claim 2, wherein: The generation of the structured business process representation by the process construction algorithm comprises: Based on the normalized entity representation and inter-entity relationship, a business process graph is created by a graph construction algorithm; The initial operation parameter is matched and bound to the corresponding operation node by a parameter binding algorithm; Based on the conditional relationship in the inter-entity relationship, a condition judgment logic is extracted by a condition expression parser to generate a condition rule set; Based on the business process graph and the condition rule set, a structured business process representation is generated by a process integration algorithm. 4.The deep learning based web business process intelligent automation execution system of claim 1, wherein: The multi-modal data comprises DOM structure data, page image data and page text content data; The processing flow of the deep learning feature extraction module comprises: The DOM structure data is encoded by a graph neural network to extract node attribute features, hierarchical relationship features, semantic role features and interaction state features, and generate a structural feature vector; The page image data is processed by a visual coding model based on an attention mechanism to extract spatial layout features, visual appearance features and element boundary features, and generate a visual feature vector; The page text content data is semantically encoded to extract text semantic features and context association features, and generate a text feature vector.

5. The deep learning based web business process intelligent automation execution system of claim 1, wherein: The processing flow of the cross-modal association fusion module comprises: The structural feature vector, visual feature vector and text feature vector are subjected to dimension alignment and feature standardization processing, and mapped to a unified feature space; The intra-modal association weight and cross-modal association weight are respectively calculated by a multi-modal attention mechanism; The modal feature vectors are weighted and aggregated based on the intra-modal association weight to generate optimized feature representations of each modality; The modal feature vectors are subjected to inter-modal feature alignment and fusion based on the cross-modal association weight, and the contribution degree of the optimized feature representations of each modality is dynamically adjusted by a gating mechanism to generate a preliminary fusion feature representation; The preliminary fusion feature representation is subjected to residual connection processing to generate a unified page feature representation. 6.A method for intelligent automation execution of a web business process based on deep learning, based on the system for intelligent automation execution of a web business process based on deep learning in any of claims 1-5, characterized in that: Comprise: Obtaining a business process description and multi-modal data of a Web page, and performing semantic analysis on the business process description to generate a structured business process representation; Deep learning feature extraction is performed on the multi-modal data to obtain corresponding structural feature vectors, visual feature vectors and text feature vectors; The structural feature vectors, visual feature vectors and text feature vectors are fused by a cross-modal association mechanism to generate a unified page feature representation; Based on the unified page feature representation, a page element intelligent recognition model is constructed to recognize and locate target elements; Based on the structured business process representation and the target element recognition result, an operation instruction sequence is generated by an instruction generation algorithm; The operation instruction sequence is executed, and the element recognition strategy and operation strategy are dynamically adjusted according to the execution feedback. 7.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is characterized in that: The processor executes the computer program to implement the steps of the deep learning-based Web business process intelligent automation execution system according to any one of claims 1-5.

8. A computer readable storage medium having stored thereon a computer program, characterized in that: The computer program is executed by a processor to implement the steps of the deep learning-based Web business process intelligent automatic execution system according to any one of claims 1-5.

Images