Data processing method and device, equipment and medium

Abstract

The embodiment of the invention discloses a data processing method and device, equipment and a medium, and is applied to the technical field of computers. The method comprises the following steps: acquiring a plurality of operation sequence groups to be clustered; performing sequence clustering on the plurality of operation sequence groups based on the operation sequences in the plurality of operation sequence groups to obtain clustering clusters related to the plurality of operation sequence groups; scene matching operation matched with the clustering scene corresponding to the clustering cluster is determined based on service operation represented by the operation sequence contained in the clustering cluster; and generating an element positioning anchor point for performing element positioning on a page element of the scene matching operation, so as to position the page element in a task page of a scene task through the element positioning anchor point when the scene task related to the clustering scene is obtained, and executing the scene matching operation of the scene task through the page element. By adopting the embodiment of the invention, the definition efficiency and flexibility of the business process can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, device and medium. Background Technology

[0002] Currently, the implementation of automated processes mainly relies on preset rules or manual configuration. For example, a search process requires manual configuration of each search step or manual writing of fixed logic code to achieve automated searching. Therefore, to automate a business process, explicit rules or templates must be predefined, and the system can only execute automated operations within a preset framework. This makes defining business processes cumbersome, inefficient, and inflexible. Summary of the Invention

[0003] This application provides a data processing method, apparatus, device, and medium that can improve the efficiency and flexibility of business process definition.

[0004] On one hand, embodiments of this application provide a data processing method, the method comprising:

[0005] Obtain multiple operation sequence groups to be clustered; multiple operation sequence groups refer to those obtained by dividing the operation sequence dataset into windows using a sliding window.

[0006] Based on the operation sequences in multiple operation sequence groups, sequence clustering is performed on the multiple operation sequence groups to obtain clusters related to the multiple operation sequence groups;

[0007] Based on the business operations represented by the operation sequences contained in the cluster, determine the scenario matching operation that matches the clustering scenario corresponding to the cluster;

[0008] Generate element positioning anchors for locating page elements in the scene matching operation. When a scene task related to the clustering scene is obtained, the element positioning anchors are used to locate page elements in the task page of the scene task, and the scene matching operation of the scene task is performed through the page elements.

[0009] On one hand, embodiments of this application provide a data processing apparatus, the apparatus comprising:

[0010] The sequence processing module is used to obtain multiple operation sequence groups to be clustered; multiple operation sequence groups refer to those obtained by dividing the operation sequence dataset into windows using a sliding window.

[0011] The sequence processing module is also used to perform sequence clustering on multiple operation sequence groups based on the operation sequences in multiple operation sequence groups, so as to obtain clusters related to the multiple operation sequence groups;

[0012] The operation processing module is used to determine the scene matching operation that matches the clustering scene corresponding to the cluster based on the business operation represented by the operation sequence contained in the cluster.

[0013] The operation processing module is also used to generate element positioning anchors for locating page elements for scene matching operations. When a scene task related to the clustering scene is obtained, the element positioning anchors are used to locate page elements in the task page of the scene task, and the scene matching operation of the scene task is performed through the page elements.

[0014] On one hand, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory is used to store a computer program, the computer program including program instructions, and the processor is configured to invoke the program instructions to execute some or all of the steps in the above method.

[0015] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, are used to perform some or all of the steps in the above-described method.

[0016] Accordingly, according to one aspect of this application, a computer program product or computer program is provided, which includes computer instructions that, when executed by a processor, can implement some or all of the steps in the above-described method.

[0017] In this embodiment, an operation sequence dataset can be obtained by collecting a series of business operation sequences, and multiple operation sequence groups can be obtained by dividing the dataset into multiple operation sequence groups through a sliding window. An operation sequence group may represent the business operation steps in a business scenario. At this time, multiple operation sequence groups can be clustered to obtain clusters. A cluster can be regarded as a potential business scenario (clustering scenario, such as a search scenario), which includes one or more similar operation sequence groups. Thus, the scenario matching operation (common steps) in the corresponding clustering scenario can be determined by the operation sequences included in the cluster. The scenario matching operation can represent the steps required to implement the clustering scenario (such as entering content in the search box, clicking the search control, etc.), that is, the execution flow of the business scenario is obtained. In addition, element positioning anchors for locating page elements for scenario matching operations can also be generated for subsequent automation process implementation. For example, when a scenario task is obtained, the page element can be located by element positioning anchor point, so that the scenario matching operation can be performed through the page element, thereby realizing automation for clustering scenarios. Therefore, by collecting the operation sequence of business operations and identifying the operation sequence, multiple business scenarios and scenario matching operations under each business scenario can be quickly determined. That is, the process judgment and configuration of business scenarios can be realized in the actual operation process of users without manual definition. This can optimize the definition steps of business processes and improve the definition efficiency and flexibility. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A schematic diagram of a network architecture provided for an embodiment of this application;

[0020] Figure 2 A schematic diagram illustrating a data processing scenario provided in an embodiment of this application;

[0021] Figure 3 A flowchart illustrating a data processing method provided in this application embodiment. Figure 1 ;

[0022] Figures 4a-4b A schematic diagram illustrating the acquisition process of an operation sequence provided in an embodiment of this application;

[0023] Figures 5a-5b A schematic diagram illustrating the identification process of a clustering scenario provided in an embodiment of this application;

[0024] Figure 6A flowchart illustrating a data processing method provided in this application embodiment. Figure 2 ;

[0025] Figures 7a-7b A schematic diagram illustrating the improvement process of the clustering scenario provided in the embodiments of this application;

[0026] Figures 8a-8e A schematic diagram illustrating the execution process of business operations provided in the embodiments of this application;

[0027] Figure 9 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0028] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] The data processing method proposed in this application is implemented in an electronic device, which can be a server or a terminal. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet computer, laptop computer, desktop computer, etc., but is not limited to these.

[0030] One of the network architecture diagrams proposed by the data processing method is as follows: Figure 1 As shown, the network architecture may include server 100 (the number of servers is not limited) and a cluster of terminal devices (the number of terminal devices is not limited, such as terminal device 200a, terminal device 200b, ..., terminal device 200n), wherein communication connections may exist between the servers. Simultaneously, a server may have a communication connection with any terminal device, so that the server can interact with the terminal device through this communication connection. The aforementioned communication connection is not limited in method; it can be directly or indirectly connected via wired communication, or directly or indirectly connected via wireless communication, or other methods, which are not limited herein. Furthermore, it is understood that the electronic devices involved in the embodiments of this application may be... Figure 1 The terminal device shown can also be Figure 1 The server shown.

[0031] It should be understood that, such as Figure 1Each terminal device in the terminal device cluster shown can be equipped with an application client for automated processing. This application client can be of any type, such as a social networking client, instant messaging client (e.g., a conferencing client), entertainment client (e.g., a live streaming client), multimedia client (e.g., a video client), information client (e.g., a news client), shopping client, or any other client capable of displaying text, images, audio, and video data. No specific type of application client is limited here.

[0032] For example, an application client refers to a client that can send and receive internet messages instantly and has information functions. Specifically, the target account's terminal device (such as...) Figure 1 Business operations can be performed on the terminal device 200a) shown. When the server detects a business operation, it can collect the operation sequence to identify possible business scenarios.

[0033] Optionally, the aforementioned terminal devices and servers can be logically separated. Therefore, when referring to terminal devices and servers below, they may be physically the same device or different devices.

[0034] For further information, please refer to [link / reference]. Figure 2 , Figure 2 This is a schematic diagram of a data processing scenario provided in an embodiment of this application. It describes how an operation sequence dataset (operation sequence a21, operation sequence a22, ..., operation sequence a2n) can be obtained. This operation sequence dataset includes multiple operation sequences, and each operation sequence corresponds to a business operation (such as a trigger operation for a control on a business page).

[0035] The operation sequence dataset can be partitioned using a sliding window to obtain multiple operation sequence groups to be clustered (operation sequence group b21, operation sequence group b22, ..., operation sequence group b24). Then, based on the operation sequences within these multiple operation sequence groups, sequence clustering can be performed to obtain related clusters (such as cluster c21, cluster c22). An operation sequence group includes at least one operation sequence.

[0036] In this cluster, each cluster represents a potential business scenario (clustering scenario), and the operation sequence groups within that cluster are similar operation sequence groups within the corresponding clustering scenario (e.g., cluster c21 includes operation sequence groups b21 and b22, and cluster c22 includes operation sequence groups b23 and b24). For example, if the clustering scenario is an e-commerce shopping scenario, the operation sequence groups obtained from the clustering can all be regarded as multiple operation processes within that e-commerce shopping scenario. For instance, the business operation represented by the operation sequence in one operation sequence group could be: entering search terms in the input box of the e-commerce platform, clicking the search control, clicking the product card on the search results page, etc.; the business operation represented by the operation sequence in another operation sequence group could be: entering search terms in the input box of the e-commerce platform, clicking the search control, scrolling through the search results page to view product prices, clicking the product card on the search results page, etc.

[0037] In other words, the purpose of clustering is to aggregate multiple groups of similar operation sequences that may fall under the same clustering scenario. This allows for the extraction of common steps within these similar operation sequence groups—that is, scenario-matching operations that correspond to the clustering scenario. For example, determining the necessary steps to achieve the desired clustering scenario through multiple operation sequence groups might include: entering search terms in an e-commerce platform's input box, clicking the search control, clicking a product card on the search results page, etc. Common steps appearing in multiple operation sequence groups can then be used as scenario-matching operations representing the execution flow of the clustering scenario.

[0038] Therefore, by collecting the operation sequence of business operations, multiple business scenarios can be analyzed and the execution flow of each business scenario can be determined. This eliminates the need for manual definition and enables rapid automation of processes.

[0039] At this point, since the execution of the scene matching operation requires triggering page elements on the business page (such as triggering a search control on the page), element positioning anchors can be generated for locating page elements for the scene matching operation.

[0040] Subsequently, when executing the automated process, the scene task related to the clustering scene can be obtained, and the page element (S21) can be located in the task page of the scene task (such as a specified control on the task page, such as the scene matching operation is the operation of clicking the search control, and the page element is the search control) through the element positioning anchor point, thereby triggering the page element to execute the scene matching operation of the scene task (S22).

[0041] Optionally, in some embodiments, the electronic device can execute the data processing method to achieve automated processes according to actual business needs. The suitable application scenarios could include vertical domain crawler automation, operation and maintenance automation, data acquisition automation, etc. This application does not limit the application scenarios; its purpose is to identify clustering scenarios through the collected operation sequences and more conveniently obtain scenario matching operations for clustering scenarios, thereby achieving automated control for clustering scenarios.

[0042] For example, in a vertical domain web scraping automation scenario, users can perform vertical domain (such as e-commerce) business operations on a business page (e.g., to check competitor product prices, they need to open a browser, enter search terms, click the search button, filter search results, view products, identify selected product data, return to the search page, flip through pages, and repeat the aforementioned steps). After collecting the operation sequence of these business operations, the scenario-matching operations for the vertical domain can be determined based on these operation sequences. Subsequently, automated web scraping tasks can be generated based on these scenario-matching operations. This allows for the periodic searching of products with specified content and the capture of the required product data. Optionally, the application of the data obtained through the web scraping task is determined according to the actual scenario. For example, in the e-commerce domain, it can be used for price analysis; in the model training domain, it can be used to obtain high-quality text corpora to train large language models, etc., without limitation.

[0043] Optionally, the large language model can be used to understand user task requirements and scene matching operations in clustering scenarios, thereby generating suitable crawling tasks. The relationship between the crawler and the corpus training of the large language model is not a one-way "crawl-feed" but is forming a positive cycle. For example, in the first stage, crawling serves training, and vertical crawling provides core corpora for training domain-specific large language models (such as medical and financial models). In the second stage, the large language model empowers crawling: it can achieve intelligent parsing, understand page semantics, more robustly handle webpage redesigns, and accurately extract main content and fields; it can also achieve task generation, automatically identifying the corresponding vertical domain based on task requirements, thereby generating corresponding scene tasks. In the third stage, after the large language model is trained, it can further guide the next round of vertical crawling, improving the intelligent effect of the automated process.

[0044] For example, in an operations and maintenance automation scenario, users can perform operations and maintenance tasks on the business page (such as operations related to collecting log data, operations related to accessing the business system to collect health status information, etc.). After collecting the operation sequence of these operations, the scenario matching operations for clustering scenarios can be determined based on these operation sequences (such as the operation steps related to collecting log data in the log analysis scenario, the operation steps related to accessing the business system and collecting health status information in the system health analysis scenario, etc.). Subsequently, automation of log analysis scenarios, automation of system health analysis, and so on can be realized.

[0045] For example, in the data collection automation scenario, users can perform business operations in the data collection scenario on the business page (such as operations related to collecting user behavior data, operations related to judging user rationality, etc.). After collecting the operation sequence of these business operations, the scenario matching operation of the clustering scenario can be determined based on these operation sequences (such as the operation steps related to collecting behavior data in the behavior analysis scenario, the operation steps related to collecting user interaction data in the user rationality analysis scenario, etc.). Subsequently, the automation of behavior analysis scenario, the automation of user rationality analysis scenario, and so on can be realized.

[0046] It is understandable that different business scenarios involve different business operations, resulting in different business data acquisition, different scenario purposes, and ultimately, different data processing operations based on the data collected from these business operations. The purpose of this application is to first determine different clustering scenarios and scenario matching operations based on the sequence of collected business operations, so that in the corresponding clustering scenario, automated processes can be achieved based on scenario matching operations and element location anchor points.

[0047] Optionally, the data involved in this application, such as business data, can be stored in a database or in a blockchain, such as through a blockchain distributed system. This application does not impose any restrictions on this.

[0048] It should be noted that in specific embodiments of this application, when scenarios involving the acquisition of user information and related data, such as collecting business data related to users, user permission or consent is required. That is, when the embodiments of this application are applied to specific products or technologies, the collection, use, and processing of relevant user data comply with the relevant laws, regulations, and standards of the relevant regions. For example, prompts can be issued through an interactive interface to indicate what data will be collected or acquired. Specifically, the types and content of this data can be presented to the user through lists or other means. Further data collection and processing will only proceed after a confirmation or instruction to allow data collection is received on the interactive interface.

[0049] It is understood that the above scenarios are merely examples and do not constitute a limitation on the application scenarios of the technical solutions provided in the embodiments of this application. The technical solutions of this application can also be applied to other scenarios. For example, as those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0050] Based on the foregoing description, this application proposes a data processing method that can be executed by the aforementioned electronic device. Please refer to... Figure 3 , Figure 3 A flowchart illustrating a data processing method provided in this application embodiment. Figure 1 Specifically, it can be Figure 1 The server shown. (As shown) Figure 3 As shown, the data processing method of this application embodiment may include the following flow:

[0051] S101. Obtain multiple operation sequence groups to be clustered.

[0052] Among them, multiple operation sequence groups refer to those obtained by dividing the operation sequence dataset into windows using a sliding window.

[0053] The operation sequence dataset includes operation sequences of multiple business operations. For example, it could be the detection of user business operations while the user is using the terminal device (or a specific application) normally, the collection of operation sequences of the business operations, and the use of the operation sequences over a period of time as the operation sequence dataset.

[0054] For example, a user performs relevant business operations in a browser, and the corresponding operation sequence is collected. Or a user performs relevant business operations on an e-commerce platform, and the corresponding operation sequence is collected. The operation sequences in this operation sequence dataset can be operation sequences collected for all applications on the terminal device, or operation sequences collected for a specific application, or operation sequences collected for a specific business within a specific application; there are no restrictions here.

[0055] For users, the technical solution of this application can collect operation sequences on terminal devices, thereby identifying the user's business scenarios and process flows in different applications. Alternatively, it can collect operation sequences from e-commerce platforms, such as e-commerce platform personnel operating the platform to perform related e-commerce business (e.g., comparing product prices, comparing popular live streams, etc.). Or, it can collect operation sequences specific to a particular business, such as a user wanting to create an automated process for public opinion monitoring on a social media platform, performing business operations related to public opinion monitoring on the social media platform and collecting the operation sequences of these operations. No further limitations are imposed here.

[0056] This could involve obtaining the operation data of a business operation of a preset type when it is detected on the current business page, and constructing an operation sequence corresponding to the business operation based on the operation data. Alternatively, the operation data can be processed according to a specified data structure to obtain the operation sequence.

[0057] The operation data includes at least one of the following: basic operation information, page information of the current business page, element information of the page element corresponding to the business operation, operation parameter information, and multimodal operation information. No further limitations are specified here.

[0058] Therefore, when a business operation is detected on the business page, the operation type can be determined. If the operation type indicates that the business operation has practical significance, the operation sequence can be collected. For example, when a triggering operation of a control causes a page redirect (such as clicking a search control) or retrieves certain data (such as downloading work logs), the operation sequence can be collected. For operations without practical significance (such as clicking a blank area), the operation sequence will not be collected.

[0059] This application allows for the complete collection of user operation sequences without the need for in-application event tracking. The operation collection engine continuously monitors user actions to collect operation sequences, providing a data foundation for subsequent scenario analysis, anchor point generation, and automated execution.

[0060] For example, such as Figures 4a-4b As shown, Figures 4a-4b This is a schematic diagram of an operation sequence acquisition scenario provided in an embodiment of this application; wherein, in Figure 4a In this context, the methods for collecting operation sequences vary depending on the application type. For example, for browser applications, relevant plugins can be installed in the browser to detect and collect operations, while for native applications, operation detection and collection can be achieved through the Accessibility API provided by the operating system.

[0061] When a business operation is detected, an operation sequence can be collected. This involves collecting operation data (which may include multimodal data), standardizing the data according to a specified data structure, and obtaining the operation sequence. This sequence can then be added to a database (such as Lance, which uses a self-developed open-source data format optimized for storing and retrieving large-scale unstructured data (such as embedded vectors and multimodal data)). The operation sequence dataset can then be retrieved from the database for scene recognition or historical query analysis.

[0062] For example, taking a browser as an example, a detection script can be injected into the business page during runtime through a browser plugin to capture all user interactions.

[0063] The preset operation types can be: navigation operations (such as page navigation, forward / backward navigation, refresh, tab switching, etc.), element interaction operations (such as click, double-click, right-click menu, hover, etc.), input operations (such as text input, dropdown menu selection, file upload, etc.), scrolling operations (such as page scrolling, element scrolling, etc.), form operations (such as submitting a form, resetting a form, etc.), and drag-and-drop operations (such as dragging elements, dragging files, etc.). The preset operation types can be configured independently and are not limited here.

[0064] For each business operation, the basic operation information collected can be: timestamp (accurate to milliseconds), operation type, triggering method (such as mouse / keyboard / touch, etc.).

[0065] The page information can include: page URL, page title, current DOM (Document Object Model) structure hash, viewport information (such as scroll position, visible area, etc.).

[0066] In this context, business operations can correspond to page elements, which are the elements triggered by the business operation (such as controls, input boxes, files, images, etc.). Optionally, for a business operation of a scrolling page type, no page elements may be required.

[0067] Therefore, when page elements exist, the corresponding element information can be obtained, such as: element tag name, element text information, element attributes (such as id, class, data-* and other core global attributes), the element's path in the DOM structure, the coordinates of the element's visible area (and / or a screenshot of the visible area), etc.

[0068] For business operations, input parameters may be involved, such as content entered in a text box or clicking an option in an option list. Therefore, the corresponding operation parameter information that can be obtained can be: the input text content (such as search terms entered in the search bar, the selected option value, the coordinates of the click, the mouse / keyboard modifier key status, etc.).

[0069] Among them, Figure 4b In the process of collecting operation data, the conditions for obtaining multimodal operation information may include at least one of the following: the business operation is a critical operation, the number of business operations between the business operation and historical business operations reaches a preset number; historical business operations refer to business operations in the last collected operation data that contain multimodal operation information.

[0070] For example, when a business operation is detected, it is determined whether the operation is critical. If it is critical, multimodal operation information is generated. If the operation is not critical, but the cumulative number of operations reaches a preset threshold, multimodal operation information is also generated. This cumulative number of operations is the number of business operations between the last time multimodal operation information was obtained and the current operation. In other words, even if the operation is not critical, multimodal operation information can still be generated when a certain number of operations are accumulated. Otherwise, if the operation is not critical and the cumulative number of operations has not reached the preset threshold, multimodal operation information is not generated; instead, the operation sequence is obtained and stored based solely on the structured information mentioned above.

[0071] The multimodal operation information includes at least one of the following: a screenshot of the current business page (which may be an original screenshot, a compressed screenshot, or a screenshot after blurring special information), text information extracted from the page screenshot, and visual features extracted from the page screenshot.

[0072] The key operations include at least one of the following: business operations of the type of ending operation (such as submitting a form or clicking a "confirm / submit / save" button); business operations of the type of abnormal operation (such as special operations such as deletion / restart); the first business operation when the current business page and the business page displayed before the current business page do not belong to the same business process (i.e., the page state has changed significantly, such as the current business page is the page for entering a new business process, which can be determined based on the domain name of the page URL, or switching from another application to a browser page, or switching to a different page tab, etc.); the first business operation after the page stay time on the current business page reaches the preset time (i.e., the first operation after the user stays on the page for more than the threshold time); and business operations when there are error prompts on the current business page (i.e., the page displays error prompts or warning messages, etc.).

[0073] In other words, for critical operations, relevant multimodal operation information can be obtained to improve the data richness and reliability of the operation sequence.

[0074] For example, taking an e-commerce shopping page as an example, the sequence of operations collected for the business operations of a user purchasing goods could be:

[0075]

[0076]

[0077] For example, in native applications, one can choose the operating system's auxiliary APIs to collect operation sequences.

[0078] The preset operation types for data collection can include the following: application management (such as starting an application, switching applications, closing applications, etc.), window operations (such as maximizing, minimizing, resizing, moving positions, etc.), menu operations (such as clicking menu items, expanding drop-down menus, etc.), control interactions (such as button clicks, text input, list selection, expanding tree nodes), clipboard operations (such as copying, pasting), and shortcut keys (such as keyboard shortcut combinations), etc., without any limitations.

[0079] The basic operational information in the collected operational data may include relevant application information, such as application name and identifier (BundleID / process name), application version number, window title, window position and size, etc.

[0080] The element information in the collected operation data may include: element type (button / text box / menu item, etc.), element identifier (Accessibility Identifier), element label and value, element path in the UI tree, element status (enabled / disabled / selected, etc.), element screenshot, etc.

[0081] The collected page information may include a snapshot of the UI tree (a tree structure representation of elements in the business page) before the operation, changes in the UI tree after the operation, and related system events (file open / save dialog boxes, etc.).

[0082] Among them, the collected operation parameter information refers to the relevant information of the parameters involved in the business operation, such as the input text, the selected file, etc.

[0083] For multimodal operation information, it can be obtained when the business operation is a critical operation, that is, the application window can be captured as a page screenshot.

[0084] Key operations can include: actions performed when a user clicks key buttons such as "Save / Submit / Delete", actions performed when a dialog box / warning box pops up, the first action performed after the application window is switched, and the first action performed when the UI state undergoes a significant change (such as when a new page has finished loading).

[0085] The methods for generating multimedia modal information include: capturing the screen image of the current application window, identifying and locating page elements within the application window, blurring special information, image compression, extracting text from the image using OCR, and generating text descriptions of the image.

[0086] For example, taking office software applications as an example, the sequence of operations collected for a user's business operation of editing a table could be:

[0087]

[0088] Operation sequences can be stored uniformly in a database to support efficient querying and analysis. For example, the data structure of an operation sequence could be:

[0089] {

[0090] Operation sequence list:

[0091] Primary key field:

[0092] operation_id: Unique ID for the operation;

[0093] session_id: Session ID (the sequence of operations within the same business process);

[0094] Time field:

[0095] timestamp: Operation timestamp;

[0096] Date partition: Date partition (YYYY-MM-DD);

[0097] Application fields:

[0098] app_type: Application type (browser / desktop);

[0099] app_name: Application name;

[0100] app_version: Application version;

[0101] page_url: Page URL (browser) or window title (native application);

[0102] Operation fields:

[0103] operation type: operation type;

[0104] target_element: Target element information (JSON);

[0105] operation params: Operation parameters (JSON);

[0106] Operation result: The result of the operation.

[0107] Context fields:

[0108] Before state hash: The hash of the state before the operation;

[0109] after state_hash: The hash of the state after the operation;

[0110] UI tree path: the path of UI elements;

[0111] Multimodal fields:

[0112] screenshot url: The path where the screenshot is stored;

[0113] OCR text: Text extracted by OCR;

[0114] visual description;

[0115] Screenshot embedding: embedding screenshot vectors;

[0116] Analysis fields:

[0117] scene id: The ID of the associated operation scene;

[0118] is_key_operation: Whether the operation is critical;

[0119] Risk level: Abnormal level (low / medium / high);

[0120] }

[0121] Among them, for operation sequences in the database (operation sequence dataset refers to the data obtained from the database for scene recognition), the data partitioning strategy can include: partitioning by date (to improve the efficiency of historical data query), partitioning by application type (to facilitate analysis for specific applications), and aggregation by session (aggregating and storing operation sequences of the same business process).

[0122] In addition, vector indexes can be built for the database. Vector indexes can be built asynchronously for the following data, supporting semantic retrieval: text description embedding of operation sequences, visual feature embedding of page screenshots, and semantic embedding of text in page screenshots.

[0123] The dataset supports interactive analysis. For example, users can query all approval operations in the internal OA system within a specific time period using natural language. In this case, the similarity between the operation sequence vectors (such as the similarity between the sequence feature vectors of the operation sequence or the descriptive feature vectors of the text description of the operation sequence and the feature vectors of the query language) can be used to identify the operation sequences that the user repeatedly performs, and to analyze high-frequency operations that can be automated.

[0124] This can be achieved by retrieving operation sequence datasets from a database, such as operation sequences for content within a specified time period or operation sequences for a specific application. Based on these operation sequence datasets, potential clustering scenarios can then be analyzed.

[0125] Among them, clustering scenarios (also known as business scenarios, operation scenarios, etc.) refer to a series of logically related operations performed by users in a specific business context to achieve a specific goal. That is, a clustering scenario refers to a set of logically related operation processes triggered by factors such as user intent, system state, and external input in a specific system context or business environment. An operation scenario includes: (1) Scenario goal: the task that the user wants to complete (such as "buying goods on an e-commerce platform"); (2) Operation sequence: the key steps to complete the task (such as searching, selecting, adding to cart, and checking out in sequence); (3) Triggering condition: the sign that the scenario starts (such as opening a specific website or clicking a specific button); (4) Ending condition: the sign that the scenario is completed (such as successful submission or the appearance of a confirmation page); (5) Target element: the UI elements involved in the operation process; (6) Operation parameters: parameters that may be different each time the operation is performed (such as search keywords, product selection, etc.).

[0126] This application can analyze potential business scenarios and determine corresponding scenario-matching operations based on the operation sequence dataset. For example, the operation sequence dataset can be segmented using a sliding window to obtain multiple operation sequence groups.

[0127] The window length of the sliding window can be determined by: the window length being determined based on a preset time, or based on the business operation frequency indicated by the operation sequence in the operation sequence dataset, or based on the operation boundary indicated by the operation sequence in the operation sequence dataset.

[0128] Taking the operation sequence dataset as an example, which includes a first operation sequence and a second operation sequence, the method to determine whether there is an operation boundary is at least one of the following: the operation application indicated by the first operation sequence is different from the operation application indicated by the second operation sequence; the time interval between the operation time indicated by the first operation sequence and the operation time indicated by the second operation sequence reaches a preset interval; the operation type indicated by the second operation sequence is an end operation type; the page domain name of the business page indicated by the first operation sequence is different from the page domain name of the business page indicated by the second operation sequence.

[0129] Specifically, when the second operation sequence is detected as an operation boundary, the second operation sequence and the operation sequences not previously included in the second operation sequence are divided into a window and grouped as an operation sequence group. For example, if the previous sliding window division ends at operation sequence a, and operation sequence b is determined to be the operation boundary, then the operation sequences between operation sequence a and operation sequence b can be grouped as an operation sequence group, meaning the division of the next sliding operation ends at operation sequence b (optionally, operation sequence b may or may not be included, depending on the determination of the operation boundary).

[0130] In other words, a sliding window can be used to segment a continuous sequence of operations into potential scenario candidates. The window segmentation strategy can include: (1) a fixed time window, such as a default window of 5 minutes; (2) dynamic window adjustment, such as adaptively adjusting the window size based on the operation density. Specifically, when operations are frequent (e.g., >10 operations per minute), the sliding window size can be shortened to 3 minutes, or when operations are sparse (e.g., <3 operations per minute), the sliding window size can be extended to 10 minutes; (3) boundary recognition, such as recognizing scenario boundaries (operation boundaries). For example, if the second operation sequence is the scenario boundary, then the sliding window is segmented up to the second operation sequence. The scenario boundary segmentation can be application switching (e.g., a user switches from one application to another, such as a user opening another social platform after shopping on an e-commerce platform; in this case, the operation sequence before the switch can be a sequence within a business scenario, and the second operation sequence may not be included in the segmentation). Alternatively, the scenario boundary segmentation can be based on the operation time interval, such as a long period of inactivity (e.g., a user has not performed any operations for more than 2 minutes); when performing business operations again, a sliding window segmentation can be performed (the second operation sequence may not be included in the segmentation). For example, scene boundaries can be defined based on a clear ending operation (such as clicking a "Submit / Complete / Close" button), which signifies the end of a business process. Therefore, a sliding window can be used for this division (the division can include a second operation sequence). Alternatively, scene boundaries can be defined by changes in the page URL domain (the division may not include a second operation sequence).

[0131] The strategy for dividing the sliding window is not limited here. It can be any of the above, or a combination of one or more of the above. For example, when dividing the sliding window based on the frequency of business operations, if there are operation boundaries, then the sliding window can also be divided. The specific strategy should be determined based on the actual scenario.

[0132] S102. Based on the operation sequences in multiple operation sequence groups, perform sequence clustering on the multiple operation sequence groups to obtain clusters related to the multiple operation sequence groups.

[0133] One operation sequence group may represent an operation in a business scenario. Multiple operation sequence groups may have the same business scenario. Therefore, multiple operation sequence groups can be clustered to realize the recurring business scenario.

[0134] For example, it could be to determine the sequence feature vector of the operation sequence in each operation sequence group to obtain the sequence feature vector of each operation sequence group; to perform sequence clustering on multiple business operation groups based on the sequence feature vectors of multiple operation sequence groups to obtain at least one cluster; and a cluster includes at least one operation sequence group.

[0135] In this case, the average feature (or concatenated vector) of the sequence feature vectors of the operation sequences in an operation sequence group can be used as the sequence feature vector of that operation sequence group.

[0136] The sequence clustering method can be any clustering algorithm, such as the K-means algorithm or the HDBSCAN (Hierarchical DBSCAN) algorithm, etc., without any limitation.

[0137] It can be understood that each cluster corresponds to a potential operation scenario, the cluster center serves as the representative operation sequence of that cluster scenario, and the size of a cluster indicates the frequency of occurrence of that operation scenario.

[0138] One way to generate a sequence feature vector for an operation sequence is to perform text encoding on text information and image encoding on image information, and then concatenate the encoded results of each operation data in the operation sequence to obtain the sequence feature vector.

[0139] For example, information such as operation type, element, and page information (e.g., application name and page URL) in the operation sequence can be encoded. Furthermore, the operation rhythm can be obtained, such as the statistical characteristics of operation intervals (mean, variance, maximum interval).

[0140] Furthermore, it is possible to generate operation description text for the operation sequence (i.e., a description of the business operation indicated by the operation sequence in natural language, such as "search for 'apple' in the search box") to obtain the semantic features of the operation sequence. For example, the operation sequence can be input into a large language model, which will output the operation description text for the operation sequence. Optionally, this operation description text can be obtained at this time, or it can be already in the operation sequence.

[0141] One way to generate sequence feature vectors is to use a multimodal encoder to generate hybrid feature vectors. For example, for structured features (operation type sequence, element type sequence), vectors are generated by a sequence encoder (LSTM / Transformer); for text features (application name, URL, element text, operation description text), vectors are generated by a text encoder; and for image features (screenshots), vectors are generated by a visual encoder. At this time, multiple vectors can be concatenated and projected onto a unified dimension through a multilayer perceptron to obtain the sequence feature vector.

[0142] S103. Determine the scenario matching operation that matches the clustering scenario corresponding to the cluster based on the business operation represented by the operation sequence contained in the cluster.

[0143] Since a cluster represents a potential business scenario, scenario description text for the corresponding business scenario can be generated based on the operation sequence contained in the cluster, so that the business scenario can be described in natural language through the operation sequence.

[0144] For example, at least one cluster includes the target cluster, and the target cluster includes at least the target operation sequence group.

[0145] Specifically, the process involves: obtaining the operation description text of the business operation indicated by the operation sequence in the target operation sequence group; generating scene prompt text based on the operation description text of the business operation indicated by the target operation sequence group and / or at least a portion of the operation sequence in the target operation sequence group; and using the scene prompt text to at least determine the scene matching operation for the clustering scene, i.e., the scene prompt text is used to generate the scene description text.

[0146] The process involves inputting the scenario prompt text into the target language model (large language model), which then extracts the key operation description text from the operation description text of the business operations indicated by the target operation sequence group based on the scenario prompt text. The key operation description text is the operation description text of the business operations that need to be performed in the clustering scenario. The business operations corresponding to the key operation description text are then identified as scenario matching operations.

[0147] For example, the scene description text may include the scene objective of the clustering scene, the operation description text of the scene matching operation, etc.

[0148] The target cluster may include one or more operation sequence groups. The scenario prompt text may include operation description text for some or all of the business operations indicated by the operation sequence groups. For example, it may be 5-10 representative operation sequences, such as a specified number of operation sequence groups that are close to the cluster center of the target cluster.

[0149] Optionally, the scenario matching operation may include common steps (or the most frequently occurring steps) in the operation sequence group within the target cluster, i.e., steps that must be performed in the clustering scenario (e.g., the step of searching for products is mandatory in a product purchase scenario). Furthermore, the scenario matching operation may also include optional steps (such as frequently occurring steps) in the operation sequence within the target cluster, such as the step of browsing product reviews.

[0150] Therefore, key operation description texts can be extracted from the operation description texts of operation sequence groups in the target cluster. These key operation description texts can be summaries of similar operation description texts in the operation sequence groups of the target cluster. For example, operation sequence group 1 in the target cluster includes operation description text 1 for entering "apple" in the search box, and operation sequence group 2 in the target cluster includes operation description text 2 for entering "fruit" in the search box. The key operation description text summarized from operation description text 1 and operation description text 2 could be entering product keywords in the search box.

[0151] Optionally, if there is only one operation sequence group, each business operation in the operation sequence group can be regarded as a key step, and the key operation description text can be extracted based on the operation description text of the business operations in the operation sequence group.

[0152] In other words, a large language model extracts key steps from the operation description text of operation sequences within an operation sequence group. These steps can be mandatory in the clustering scenario (e.g., steps appearing in every operation sequence group) or optional (e.g., steps appearing in most operation sequence groups). Therefore, the large language model can identify similar operation description texts between operation sequence groups of a target cluster based on the operation description texts within the operation sequence groups, determine whether these can be considered key steps, and then generate key operation description texts based on these similar operation description texts.

[0153] For example, scenario prompt text can be constructed based on prompt templates and required data. The required data may include 5-10 representative operation sequences (in natural language) from scenario clusters, application names and page URLs involved in the operation sequences, page screenshots of the operation sequences, operation parameters, and other data, which can be determined according to the actual scenario.

[0154] The generated scene prompt text can be:

[0155] {

[0156] You are an operation scenario analysis expert. I will give you a sequence of user actions on a computer. Please analyze and describe what the user is doing.

[0157] Input information:

[0158] Operation sequence example 1:

[0159] At 10:25:30, type "apple" into the search box on the homepage;

[0160] 10:25:32 Click the search button;

[0161] At 10:25:35, click on the third product on the search results page;

[0162] 10:25:40 Scrolling through comments on the product details page;

[0163] 10:25:45 Clicked the "Add to Cart" button;

[0164] Operation sequence example 2:

[0165] At 14:30:15, enter "fruit" in the search box on the homepage;

[0166] 14:30:18 Click the search button;

[0167] At 14:30:22, clicked on the first product on the search results page;

[0168] At 14:30:28, click the "Buy Now" button on the product details page;

[0169] ...(More examples, i.e., the operation description text of the business operations in some operation sequence groups)

[0170] Please answer the following questions:

[0171] 1. What is the user's operational goal? (Describe it in one sentence)

[0172] 2. What are the key steps in this operation scenario? (List them in order, including mandatory steps and optional steps)

[0173] 3. What parameters might change in each operation? (i.e., the operational parameters involved)

[0174] 4. Under what circumstances is this scenario typically triggered?

[0175] 5. What marks the completion of this scene?

[0176] Please output the results in JSON format.

[0177] }

[0178] Therefore, large language models can answer the above questions sequentially based on the operational description text, for example, the output result would be:

[0179] {

[0180] "scene_goal": "Search and purchase goods on an e-commerce platform" (action objective);

[0181] "confidence score": 0.9;

[0182] "key_steps": [(key steps);

[0183] “step_order”: 1;

[0184] "step_description": "Enter product keywords in the search box";

[0185] "operation_type": "text input";

[0186] “variable_params”: [“search keywords”] (parameters that may vary);

[0187] }

[0188] {

[0189] “step_order”: 2;

[0190] "step_description": "Click the search button";

[0191] "operation_type": "click";

[0192] }

[0193] {

[0194] “step_order”: 3;

[0195] “step_description”: “Select a product in the search results”;

[0196] "operation_type": "click";

[0197] “variable_params”: [“selected product location”;

[0198] }

[0199] {

[0200] “step_order”: 4;

[0201] "step_description": "View product details and reviews";

[0202] "operation_type": "scroll";

[0203] "is_optional": true;

[0204] },

[0205] {

[0206] “step order”: 5;

[0207] "step_description": "Add to cart or buy now";

[0208] "operation_type": "click";

[0209] “variable_params”: [“Purchase method”];

[0210] }

[0211] {

[0212] "trigger_conditions": [(trigger conditions)]

[0213] "Open the homepage of the e-commerce website";

[0214] "When you need to purchase a specific item"; ]

[0216] "end_conditions": [(Completion flags)]

[0217] "Item successfully added to cart";

[0218] "Proceed to the checkout page"; ]

[0220] }

[0221] Therefore, the large language model can understand the operation description text of the operation sequence in the operation sequence group, so as to extract the required steps and optional steps in sequence (that is, extract the key step description text of the key steps). It can be understood that the business process under this clustering scenario can be realized by executing the scenario matching operations in the order of the scenario matching operations.

[0222] Optionally, when outputting the description text of key steps, the importance of the key steps will also be output, with higher importance for required steps and lower importance for optional steps.

[0223] For example, the key steps extracted from similar e-commerce shopping operation sequences in a cluster within a cluster could be:

[0224]

[0225] It's understandable that if the operation sequence dataset contains operation sequences from multiple business scenarios, clustering can be used to obtain clustered clusters, and then scene description text can be generated to obtain the clustered scenarios and scene matching operations. If the operation sequence dataset itself only targets a single business operation, then a large language model can be used directly to obtain the clustered scenarios and scene matching operations. For example, after a user performs a web crawling operation in a vertical domain, when generating a web crawling process based on the operation sequence obtained from that operation, this part of the operation sequence can be directly obtained to generate scene matching operations. No further limitations are imposed here.

[0226] S104. Generate element positioning anchors for locating page elements in the scene matching operation, so that when a scene task related to the clustering scene is obtained, the page elements are located in the task page of the scene task through the element positioning anchors, and the scene matching operation of the scene task is performed through the page elements.

[0227] For scene matching operations involving page elements, element positioning anchors need to be generated to locate these page elements. Since a scene matching operation is a business operation represented by a determined key operation description text, it is not an actual business operation but rather extracted from similar operation description texts of similar operation sequences within one or more operation sequence groups. Therefore, the page element indicated by the operation sequence corresponding to any similar operation description text can be used as the page element for that scene matching operation.

[0228] For example, operation sequence group 1 in the target cluster includes operation description text 1 for entering "apple" in the search box, and operation sequence group 2 in the target cluster includes operation description text 2 for entering "fruit" in the search box. The key operation description text summarized from operation description text 1 and operation description text 2 can be the product keyword entered in the search box. At this time, the element information related to the page element in the operation sequence corresponding to operation description text 1 can be used as the element information of the page element of the key operation description text (i.e., the search box).

[0229] Therefore, element positioning anchors (virtual anchors) can be generated for each key step in the scenario by clustering the page elements involved. For example, the element positioning anchor ID for the search box is VA_[hash1], the element positioning anchor ID for the search button is VA_[hash2], the element positioning anchor ID for the product card is VA_[hash3], the element positioning anchor ID for the add-to-cart button is VA_[hash4], and so on. These element positioning anchors enable accurate location of page elements during subsequent automated execution, and even if the page is redesigned, the corresponding page elements can be found through multiple positioning strategies.

[0230] In other words, in clustering scenarios, when a scenario matching operation needs to be performed, the page element is located on the current task page through the element positioning anchor point, so as to trigger the scenario matching operation by triggering the page element.

[0231] For example, such as Figures 5a-5b As shown, Figures 5a-5b This application provides a schematic diagram of a clustering scenario recognition process; wherein, in Figure 5a In this process, the operation sequence dataset is divided into operation sequence groups using a sliding window. These operation sequence groups are then clustered to obtain clusters. If a cluster is an existing cluster (i.e., a clustering scenario has already been determined), it is ignored. If a cluster is a new cluster, it represents a new clustering scenario. At this point, a scenario description text is generated based on the operation sequence groups within the clustering scenario. This scenario description text includes the extracted scenario target, scenario matching operation, and operation parameters. Finally, element positioning anchors for the page elements of the scenario matching operation are generated.

[0232] In addition, the scene confidence of the clustering scene can be calculated at this time to determine whether the scene confidence is greater than or equal to the preset confidence. If the scene confidence is greater than or equal to the preset confidence, the scene description text of the clustering scene can be added to the scene library. If the scene confidence is less than the preset confidence, the clustering scene can be determined as a scene to be confirmed to trigger user verification.

[0233] Furthermore, user feedback on the clustering scenario can be determined based on the user's actual operations, and the clustering scenario can be updated based on this feedback. Specific steps are detailed in the following embodiments.

[0234] Specifically, for element positioning anchors, positioning information for locating page elements can be generated based on the operation sequence of scene matching operations, and positioning strategies can be generated for the positioning information. The positioning information and positioning strategies are used as element positioning anchors; the positioning strategy is used to indicate the positioning priority of multiple positioning information.

[0235] The positioning strategy includes at least one of the following: state features, element features, and element description text; the state features are obtained through the page information of the business page where the scene matching operation is located, and the element features and element description text are obtained through the element information of the page elements of the scene matching operation.

[0236] In other words, Figure 5bIn this process, when a scene matching operation is determined, the operation sequence of the scene matching operation can be obtained. Page information can be extracted from the operation sequence to generate the state features of the business page and generate a state anchor ID. At this time, element features can be generated based on some or all of the element information, and element description text can be generated based on some or all of the element information. At this point, multiple positioning strategies can be generated based on this positioning information, that is, each positioning information corresponds to a positioning priority.

[0237] In addition, it can determine whether a page element is a key element (i.e., an element that requires user confirmation before being triggered, such as a payment control) and mark the page element as a key element or a regular element. At this time, the element's positioning anchor point can be stored in the anchor point library and associated with the corresponding scene matching operation in the clustered scenes in the scene library.

[0238] Therefore, page elements can be located on the task page according to their location priority, based on multiple location information, until the page element is located.

[0239] Element positioning anchors are stable identifiers generated for target elements, used to accurately locate the page elements in subsequent automated operations. Virtual anchors can generate stable identifiers by comprehensively considering multi-dimensional features and provide multiple positioning strategies, significantly improving the accuracy and robustness of element positioning.

[0240] Element positioning anchors are stable identifiers generated for target elements, used to accurately locate the page elements in subsequent automated operations. Virtual anchors can generate stable identifiers by comprehensively considering multi-dimensional features and provide multiple positioning strategies, significantly improving the accuracy and robustness of element positioning.

[0241] The data structure for element positioning anchors can be:

[0242] {

[0243] anchor_id: A unique identifier for the anchor point;

[0244] anchor_type: Anchor type;

[0245] app_identifier: Application identifier (URL domain / Bundle ID);

[0246] element_semantic: Element semantics (e.g., "search button");

[0247] element_role: The function role of the element (such as "submit form");

[0248] generation_context: Generation context;

[0249] generation_time: Generation time;

[0250] app_version: Application version;

[0251] user_operation: The user operation that was triggered;

[0252] visual_features: visual features;

[0253] screenshot_hash: The hash of the element's screenshot;

[0254] ocr_text: The text recognized by OCR;

[0255] visual_embedding: Visual feature vector;

[0256] context_features: Contextual features;

[0257] parent_element: The anchor point of the parent element;

[0258] sibling_elements: Anchor points of adjacent elements;

[0259] page_layout: Page layout fingerprint;

[0260] locator_strategies: A list of locator strategies (sorted by priority);

[0261] strategy_1: Highest priority strategy;

[0262] strategy_2: Second-priority strategy;

[0263] strategy_n: fallback strategy;

[0264] metadata:

[0265] is_key_element: Whether the element is key (the element that needs to be HITLed);

[0266] risk_level: Operational anomaly level;

[0267] usage_frequency: Frequency of use;

[0268] }

[0269] The functions of element positioning anchors include: accurate positioning (accurately finding the corresponding element in different versions of the application through multiple positioning strategies), playback operation (supporting reliable playback of operation sequences), cross-version adaptation (automatically adapting to new element positioning after application redesign), and generating scene tasks (providing stable references for page elements in scene tasks).

[0270] Among them, the state features can be (1) URL features, such as complete URL + query parameters + hash fragment; (2) DOM structure hash (the tree object model generated by the browser when parsing the HTML document, which is the core operation target of front-end automation), such as the DOM structure signature of the main area of ​​the page; (3) visible text content, such as the hash of the visible text on the page (used to detect content changes); (4) dynamic state, such as Tab active state, expanded / collapsed state, and form filling state; (5) layout fingerprint, such as the relative position relationship of the main area; (6) anchor point unique identifier, such as using the SHA-256 hash algorithm to generate an ID by combining the above features.

[0271] For element features, they can be (1) structural features, such as element label name (e.g., button, input), element ID attribute (if it exists and is not dynamically generated), element class attribute (filter out dynamic classes), custom data-* attributes, and the path of the element in the DOM tree; (2) content features, such as element text content, element aria-label attribute, element placeholder attribute, and element title attribute; (3) visual features, such as screenshot of the visible area of ​​the element, text recognized by OCR, visual attributes such as element color and shape, and the relative position of the element (relative position from the top / left of the page); (4) context features, such as parent element features, adjacent sibling element features, layout features of the area, and extracting multi-dimensional features for the target element operated by the user.

[0272] For (3) visual features, the matching process is as follows: 1. Take a screenshot of the current page; 2. Search for areas similar to page elements in the screenshot (using template matching algorithm); 3. Calculate the visual similarity (cosine similarity) of candidate areas; 4. Combine OCR text matching to select the most similar area; 5. Locate the center point of the area to locate the page element.

[0273] This involves determining different location priorities for different location information to construct a location strategy; these priorities can be pre-configured. For example:

[0274] Priority Strategy type illustrate 1 data-* Attribute Location Use stable attributes such as data-testid 2 ID attribute positioning Use element ID 3 Stable CSS class positioning Use semantic classes 4 Text content positioning Locating by element text 5 ARIA label positioning Use accessibility attributes 6 Relative position positioning Relative to the position of a stable parent element 7 XPath positioning Full DOM path 8 Visual positioning Matching via screenshot

[0275] This includes generating element description text for page elements using a large language model. For example, it can generate element tooltip text, input the tooltip text into the large language model, and output the element description text. For example:

[0276] {

[0277] You are a web UI analytics expert. Please analyze the semantics and functionality of the following page elements.

[0278] Element information:

[0279] HTML tags: <button class＝"...”id＝"...”> search< / button> ;

[0280] Element text: "Search";

[0281] Element screenshot: [image];

[0282] Location: xxx.com;

[0283] User action: Click;

[0284] Please answer:

[0285] 1. What is the main function of this element?

[0286] 2. What happens after the user clicks?

[0287] 3. What is the role of this element in the business process?

[0288] 4. Describe this element with a short semantic tag.

[0289] Please output in JSON format.

[0290] }

[0291] The resulting element description text can be:

[0292] {

[0293] “element_function”: “Trigger product search”;

[0294] "expected_behavior": "Submit keywords in the search box and jump to the search results page";

[0295] "business_role": "Key trigger button for the search process";

[0296] "semantic_label": "Product search trigger button";

[0297] }

[0298] Optionally, for page elements, it is also possible to determine whether they are key elements. For example, the rules for determining key elements are: (1) elements with high abnormal operation, such as text containing special words such as: delete, clear, restart, close, submit, purchase, payment, etc., and buttons with irreversible operation consequences and buttons involving transactions, as determined by the large model; (2) core business elements, such as form submission buttons, payment / settlement buttons, confirmation / cancel buttons, file upload / download buttons; (3) frequently operated elements, such as elements whose user operation frequency is greater than the threshold, elements that appear in multiple operation scenarios, etc. No restrictions are imposed here.

[0299] This could mean that all key elements require manual confirmation, or that only some key elements require manual confirmation, such as key elements with low impact on users that do not require manual confirmation. No specific limitation is made here.

[0300] Examples of key element annotations for page elements include:

[0301]

[0302] For native applications, after collecting the operation sequence (including element trees) based on the operating system's auxiliary system API, positioning information and positioning strategies for locating page elements can be generated based on the relevant data in the operation sequence, thereby obtaining element positioning anchor points.

[0303] The data structure for the obtained element positioning anchor points can be:

[0304] {

[0305] Unified Element Model:

[0306] element_type: Element type (button / textbox / menu / list / …);

[0307] element_identifier: Element identifier (Accessibility ID);

[0308] element_label: Element label (visible text);

[0309] element_value: The element value (such as the content of a text box);

[0310] element_state: Element state (enabled / disabled / checked / ...);

[0311] element_bounds: element boundaries (x, y, width, height);

[0312] element_path: The path of the element in the UI tree;

[0313] platform_specific: Platform-specific attribute;

[0314] }

[0315] The positioning strategy can be:

[0316] Priority Strategy type illustrate 1 Accessibility Identifier 2 Label+Type Tag text + element type combination 3 UI tree path The complete path from the root node to the target 4 relative position Relative to the position of a stable parent element 5 Visual positioning Screenshot matching + OCR

[0317] In this embodiment, an operation sequence dataset can be obtained by collecting a series of business operation sequences, and multiple operation sequence groups can be obtained by dividing the dataset into multiple operation sequence groups through a sliding window. An operation sequence group may represent the business operation steps in a business scenario. At this time, multiple operation sequence groups can be clustered to obtain clusters. A cluster can be regarded as a potential business scenario (clustering scenario, such as a search scenario), which includes one or more similar operation sequence groups. Thus, the scenario matching operation (common steps) in the corresponding clustering scenario can be determined by the operation sequences included in the cluster. The scenario matching operation can represent the steps required to implement the clustering scenario (such as entering content in the search box, clicking the search control, etc.), that is, the execution flow of the business scenario is obtained. In addition, element positioning anchors for locating page elements for scenario matching operations can also be generated for subsequent automation process implementation. For example, when a scenario task is obtained, the page element can be located by element positioning anchor point, so that the scenario matching operation can be performed through the page element, thereby realizing automation for clustering scenarios. Therefore, by collecting the operation sequence of business operations and identifying the operation sequence, multiple business scenarios and scenario matching operations under each business scenario can be quickly determined. That is, the process judgment and configuration of business scenarios can be realized in the actual operation process of users without manual definition. This can optimize the definition steps of business processes and improve the definition efficiency and flexibility.

[0318] Based on the foregoing description, this application proposes a data processing method that can be executed by the aforementioned electronic device. Please refer to... Figure 6 , Figure 6 A flowchart illustrating a data processing method provided in this application embodiment. Figure 2 Specifically, it can be Figure 1 The server shown. (As shown) Figure 6 As shown, the data processing method of this application embodiment may include the following flow:

[0319] S201. Obtain multiple operation sequence groups to be clustered. Based on the operation sequences in the multiple operation sequence groups, perform sequence clustering on the multiple operation sequence groups to obtain clusters related to the multiple operation sequence groups.

[0320] S202: Based on the business operations represented by the operation sequences contained in the cluster, determine the scene matching operation that matches the clustering scenario corresponding to the cluster, and generate element positioning anchors for locating page elements for the scene matching operation. Specific implementations of S201-S202 can be found in the relevant operations of the above embodiments, and will not be repeated here.

[0321] S203. Determine the scene confidence of the scene matching operation. If the scene confidence is greater than or equal to the preset scene confidence, add the scene matching operation to the scene library and add the element positioning anchor point to the anchor point library.

[0322] Once the clustering scenario is determined, that is, after the scenario description text containing scenario matching operations (such as operation description text) is determined, the scenario confidence of the clustering scenario can be determined to decide whether to add the clustering scenario to the scenario library for subsequent automated processes.

[0323] For example, if the scenario confidence level is greater than or equal to a preset scenario confidence level, the clustering scenario is determined to be trustworthy, and the scenario description text of the clustering scenario is added to the scenario library; the scenario library is used to execute the scenario tasks of the clustering scenario. At the same time, the element positioning anchor points of the page elements of the scenario matching operation in the clustering scenario are added to the anchor point library.

[0324] Optionally, if the scene confidence is less than the preset scene confidence, the scene matching operation is added to the scene library or the candidate library after user confirmation.

[0325] For example, confidence levels can be categorized as follows: for high confidence (>0.8): directly store in the scenario library and recommend to users for automation; for medium confidence (0.6-0.8): store in the candidate library, wait for user verification before storing in the scenario library and recommending for automation; for low confidence (<0.6): store in the candidate library, require explicit user confirmation before adding to the scenario library and recommending for automation.

[0326] Specifically, determining the scene confidence score for scene matching operations can involve: determining the feature similarity between the sequence feature vectors of each operation sequence group in the target cluster, and determining a first confidence score based on the feature similarity; and / or,

[0327] The frequency of occurrence of the scene matching operation in the business operations represented by each operation sequence group is determined, and a second confidence level is determined based on the frequency of occurrence; and / or, when the scene matching operation is determined by the target model based on the operation sequences in the target operation sequence group, the model confidence level output by the target model for the scene matching operation is determined as the third confidence level. In other words, one or more confidence levels can be determined.

[0328] The scene confidence can be determined based on at least one of the following: first confidence (cluster similarity), second confidence (heuristic confidence), and third confidence (confidence output by LLM). For example, the scene confidence can be obtained by weighted summation of multiple confidence levels.

[0329] For example, scenario confidence = w1 × first confidence + w2 × second confidence + w3 × third confidence.

[0330] Where w1, w2, and w3 are weights, and the sum of w1, w2, and w3 is 1.

[0331] Here, the first confidence level refers to the average similarity of the features among the various operation sequence groups in the target cluster. This allows us to determine the feature similarity between any two operation sequence groups, thus obtaining the average similarity of all features.

[0332] The second confidence level can refer to the average frequency of the scene matching operation in each operation sequence group. For example, the frequency of each scene matching operation in each operation sequence group can be determined sequentially to obtain the average frequency. Furthermore, the presence of the scene matching operation in an operation sequence group can be determined based on the textual similarity between the operation description text of the scene matching operation and the operation description text of an operation sequence in that group.

[0333] The third confidence level is the confidence level obtained synchronously by the large language model when outputting scene description text.

[0334] Specifically, when a scenario matching operation is added to the candidate library, the clustering scenarios in the candidate library can be validated before being added to the scenario library, i.e., adaptive improvement of the clustering scenarios. For example, when a user performs a business operation related to a clustering scenario, the system predicts the user's next business operation in the clustering scenario based on the scenario matching operation and the user's historical business operations. The predicted next business operation is an operation in the scenario matching operation. If the user's current business operation is the same as the predicted next business operation, the scenario confidence of the scenario matching operation is increased until the adjusted scenario confidence is greater than or equal to the preset scenario confidence. If the user's current business operation is different from the predicted next business operation, the scenario confidence of the scenario matching operation is decreased.

[0335] In other words, based on the scene matching operation in the clustering scenario, the user's next business operation is predicted. When the user's actual operation is the same as the predicted next business operation, it indicates that the clustering scenario has a certain reliability, which can improve the scene confidence of the clustering scenario. When the scene confidence reaches the preset scene confidence, the clustering scenario can be added from the candidate library to the scene library.

[0336] For example, if the user's actual action differs from the predicted next business action, the scenario confidence level should be reduced. Optionally, the scenario matching operation in the clustering scenario can also be adjusted, such as adding the user's actual action to the scenario matching operation, or making the scenario matching operation an optional step. No limitations are specified here.

[0337] The predicted next business operation refers to the operation selected from the scenario matching operation. For example, if the user's business operation is to click on a product card, the predicted next business operation is to browse product reviews, click the "Add to Cart" button, or click the "Buy Now" button on the product details page.

[0338] If the predicted next business operation is to browse product reviews, but the actual operation is to click the "Add to Cart" button, and the "Add to Cart" button is clicked after browsing product reviews, then browsing product reviews is an optional step (in this case, the corresponding operation can be adjusted to an optional step in the scenario matching operation).

[0339] If the actual operation is not any of the operations in the scenario matching operation, such as adding a store to enter the store homepage, it means that the actual operation is a new step that can be executed in this clustering scenario. Therefore, the actual operation can be added to the scenario matching operation and configured as an optional step.

[0340] Similarly, for clustering scenarios in the scenario library, the scenario adaptive improvement algorithm can also be implemented. For example, if the scenario matching operation is added to the scenario library, when the user performs a business operation related to the clustering scenario, the next business operation of the user in the clustering scenario is predicted based on the scenario matching operation and the user's historical business operations. The predicted next business operation is the operation in the scenario matching operation. If the business operation currently performed by the user is different from the predicted next business operation, the scenario matching operation is adjusted (or the confidence level can be adjusted simultaneously).

[0341] Determining whether a user has performed a business operation related to a clustering scenario can be done by assessing the user's past business operations (e.g., five previously performed business operations) to determine if a matching clustering scenario exists. In other words, when refining a clustering scenario, the user's actual actions can be detected to determine the current clustering scenario and refine the process.

[0342] The initial clustering scenario identification may not be entirely accurate. This system continuously refines the scenario definition through subsequent user actions. The refinement mechanism refers to the prediction of the next action and scenario optimization. In other words, when a user performs a business operation, the system predicts the next action in real time and refines the scenario based on the difference between the user's actual action and the prediction. The overall process is a closed loop of prediction, verification, and adjustment: user action → BST prediction → comparison and verification → scenario adjustment → BST update → continued prediction. This closed-loop mechanism ensures continuous optimization of the scenario definition and prediction model, ultimately converging to a stable state.

[0343] When a user begins a series of business operations, the system's real-time matching strategy is as follows: calculate the similarity between the operation sequence of historical business operations (such as operation description text) and known clustered scenarios (such as scenario description text). If the highest similarity is greater than a preset threshold, the match is considered successful. If multiple scenarios have similarities, the most recently used clustered scenario is selected.

[0344] In this context, predicting the next business operation can be achieved by using BST (Behavior Sequence Transformer) encoding and prediction, such as using BST to model and predict the operation sequence.

[0345] Among them, the BST features are constructed as follows: (1) Operation type embedding: the operation type (click / input / scroll) is encoded as a vector; (2) Element type embedding: the element type (button / textbox / link) is encoded as a vector; (3) Time position embedding: the time information in the operation sequence is encoded; (4) Time interval embedding: the time interval between adjacent operations is encoded.

[0346] The prediction process involves inputting the encoded features of historical operation sequences and scene descriptions of similar clustered scenarios into the BST model to output the predicted next operation. The BST model can also learn online; for example, the difference between the predicted next step and the actual step can be used to optimize the BST model. Examples include negative sample suppression (reward = -1) for incorrect predictions, positive sample reinforcement (reward = +1) for actual operations, and contrastive learning (simultaneously learning "what should not be done" and "what should be done").

[0347] The scenario version management and persistence feature ensures that all clustering scenario updates are persisted to the scenario library, supporting version management and rollback, as well as retrieval, reuse, and evolution. Version management strategies may include creating a new version for each major adjustment (addition / deletion steps, adjustment order), retaining historical versions for rollback and A / B comparison, marking the applicable scope of each version (application version, page version, etc.), and recording the training status and prediction accuracy of the BST model.

[0348] The scene index structure is as follows:

[0349] {

[0350] Scenario knowledge base index;

[0351] Scene Feature Vector Index (HNSW);

[0352] Used for quickly retrieving similar scenarios;

[0353] Application-Scene Mapping Index;

[0354] Quickly find relevant scenarios by application name;

[0355] User-Scenario Mapping Index;

[0356] Record the scenarios that each user frequently uses;

[0357] Time series index;

[0358] Query the evolution history of a scene by time range;

[0359] }

[0360] The scenario reuse mechanism refers to the ability of a user to reference a similar scenario in application B when operating in application A. For example, cross-application scenario migration is as follows: (1) Calculate the feature vector of the current operation sequence in application A; (2) Search for similar scenarios in the scenario library (not limited to application A); (3) If a highly similar scenario is found in application B (similarity > 0.8); (4) Call the agent to translate the scenario in application B to application A; (5) Replace the element anchor (element in application B -> corresponding element in application A); (6) Create a new clustering scenario for application A.

[0361] For example, a user's shopping experience on e-commerce platform A can be migrated to the shopping experience on e-commerce platform B. The key is to identify page elements with the same function in different applications.

[0362] Alternatively, one application of the scenario library could be to detect user business operations in real time and prompt the user when the user's next business operation is predicted, thereby improving the convenience of the user in performing business operations (such as whether to click the search control, and automatically locating and triggering the search control after confirmation, without requiring the user to search for it manually).

[0363] For example, when a user performs a business operation related to a clustering scenario, the system predicts the user's next business operation in the clustering scenario based on the scenario matching operation and the user's historical business operations. The predicted next business operation is the operation in the scenario matching operation. The system generates operation prompt information for the predicted next business operation and displays the operation prompt information on the business page of the clustering scenario. After the user confirms the operation prompt information, the system obtains the element positioning anchor point of the predicted next business operation. The system then locates the page element of the predicted next business operation on the business page through the element positioning anchor point, so as to execute the predicted next business operation through the located page element.

[0364] For example, such as Figures 7a-7b As shown, Figures 7a-7b A schematic diagram illustrating the improvement process of the clustering scenario provided in this application embodiment; wherein, in Figure 7a In the process, S711, the user performs a business operation (such as searching for products); S712, when the server detects the business operation, it can query the scene library for matching clustered scenes, such as returning the "e-commerce shopping" scene (e.g., matching degree greater than 80%); S713, the server can predict the next business operation based on the user's historical business operations (e.g., obtaining a certain number of business operation operation sequences) and the scene matching operation of the clustered scene; S714, if the predicted next business operation is clicking the search control, the server prompts the user "Do you want to click the search button?"

[0365] Among them, Figure 7b In the example, improvements to the clustering scenario could be as follows: S721, the user performs a business operation (searching for products); S722, when the server detects the business operation, it can query the scenario library for matching clustering scenarios, such as returning the "e-commerce shopping" scenario (e.g., matching degree greater than 80%); S723, the server can predict the next business operation based on the user's historical business operations (e.g., obtaining a certain number of business operation operation sequences) and the scenario matching operation of the clustering scenario; S724, wait for the user's actual operation, such as the user clicking the search control (matching the prediction, i.e., the user performs the predicted operation); S725, the server can increase the scenario confidence of the clustering scenarios in the scenario library; or, S726, wait for the user's actual operation, such as the user clicking the filter condition (not matching the prediction, i.e., the user performs other operations); S727, the server treats "clicking the filter" as a new step, i.e., collecting the operation sequence to obtain the operation description text and generating element positioning anchors for locating page elements; S728, the server updates the clustering scenarios in the scenario library, such as updating the scenario description text based on the operation description text and adding the element positioning anchors to the anchor library, etc.

[0366] S204. When the task requirements submitted by the user are obtained, determine the clustering scenario associated with the task requirements and obtain the scenario matching operation of the clustering scenario from the scenario library.

[0367] The scene matching operation is stored in a scene library, and the element positioning anchor points are stored in an anchor point library. Users can submit task requirements (such as "search for the keyword 'apple' on platform X"), at which point the clustering scene associated with the task requirement can be determined. For example, similar clustering scenes (such as "search scene") can be determined based on the text similarity between the task requirement and the scene description text of the clustering scene.

[0368] At this point, you can obtain the relevant scene matching operations from the scene description text of the clustered scene in the scene library (this is a mandatory step). In addition, you can obtain the anchor ID of the page element involved in the scene matching operation (used to obtain the corresponding element positioning anchor from the anchor library), as well as the operation parameters involved (such as the input keyword as an operation parameter, which is specified by the task requirements).

[0369] S205. Generate scene tasks related to the clustering scene based on scene matching operations and task requirements.

[0370] Specifically, a corresponding scenario task can be generated based on the scenario matching operation and the operation parameters in the task requirements. The scenario task includes specific task steps, which include scenario matching operations combined with the operation parameters.

[0371] For example, if the scene matching operation is "enter a keyword in the input box", and the operation parameter indicates that the keyword is "apple", then the generated task steps will be "enter 'apple' in the input box".

[0372] Among these, scene tasks can be generated using large language models.

[0373] S206. When performing a scene matching operation on the task page according to the scene task, obtain the element positioning anchor from the anchor point library, and locate the page element on the task page through the element positioning anchor.

[0374] Executing a scenario task is equivalent to executing task steps. Therefore, when executing a task step on a task page, an element positioning anchor can be retrieved from the anchor library according to the anchor ID corresponding to the scenario matching operation. This allows locating page elements (such as a search box) on the task page. At this point, task steps (i.e., scenario matching operations) can be executed based on these page elements. For example, triggering the search box and entering "apple" completes the current task step, and the process continues to the next task step. In other words, the various operations within a scenario matching operation have an execution order, determined by the order of operations indicated by the operation sequence group within the cluster.

[0375] The element positioning anchor includes multiple positioning information and a positioning strategy for locating page elements. The positioning strategy indicates the positioning priority of the multiple positioning information. Locating a page element can be done by sequentially locating the page element on the task page based on the multiple positioning information according to their positioning priority. If the page element is located using the target positioning information among the multiple positioning information, the positioning priority of the target positioning information is adjusted. If the page element is not located using any of the multiple positioning information, an element confirmation message is generated so that the user can confirm the page element on the task page. The confirmed page element can then be used to update the element positioning anchor.

[0376] In other words, if any target location information locates a page element, the location priority of the target location information can be adjusted (such as increasing the location priority) while executing the task steps, which can improve the location flexibility and efficiency of page elements.

[0377] If multiple location information fails to locate a page element, manual confirmation is required. In this case, the element's positioning anchor point can be updated based on the confirmed page element. For example, the element information of the confirmed page element can be used to directly update the relevant information in the original element's positioning anchor point, or a new element's positioning anchor point can be added. This way, when locating a page element next time, multiple element positioning anchor points can be used for accurate positioning.

[0378] Alternatively, the element positioning anchor can define the application corresponding to a page element. When locating a page element later, the corresponding element positioning anchor can be found from the anchor library based on the application of the current task page to achieve element positioning. In other words, the execution logic of one clustering scenario can be applied to similar business scenarios in multiple applications, such as product search scenarios on different e-commerce platforms. This allows for automated processes on a specific platform and migration of business scenarios between different platforms with only the same clustering scenario and corresponding adapted element positioning anchors.

[0379] Specifically, since it is determined whether a page element is a key element when generating element positioning anchors, when performing task steps through page elements, if the element positioning anchor indicates that the page element is a key element, a trigger prompt message is generated; after the user confirms the trigger prompt message, the page element is triggered to perform the scene matching operation.

[0380] The task requirements can also include location trigger conditions (such as executing once a week at a fixed time). In this way, when implementing automated processes, in addition to automating a single task, multiple tasks can also be automated. That is, when the location trigger conditions are detected, the scenario task is automatically executed.

[0381] This solution involves collecting user operation sequences to identify and cluster scenarios, facilitating subsequent automated processes. Optionally, before executing the automated process, the clustered scenarios can be refined, such as by continuously optimizing them based on actual user operations to obtain a reliable scenario library. This library can then be used for automated processes or real-time user operation prediction. Furthermore, clustered scenarios can be identified at fixed intervals to add new business scenarios to the scenario library. Alternatively, the clustered scenarios can be refined at fixed intervals, thus achieving a cyclical process of building, refining, and using clustered scenarios, improving overall effectiveness.

[0382] For example, such as Figures 8a-8e As shown, Figures 8a-8e This is a schematic diagram illustrating the execution process of a business operation provided in an embodiment of this application; wherein, in Figure 8a In this process, the location of page elements can be as follows: S811, the server retrieves the element location anchor from the anchor database based on the anchor ID. This element location anchor includes location information and a location strategy; S812, the server locates page elements on the task page according to the location priority in the location strategy, based on the location information; S813, if the page element is successfully located using the target location information, the server records that the target location information has been successfully located; S814, the server increases the location priority of the target location information in the location strategy in the anchor database; or, S815, if location fails using all location information, the server records that the anchor location has failed; S816, the server requests manual confirmation; S817, after manual confirmation of the page element, the server collects relevant information to analyze the location information of the new element, and then generates a new element location anchor and updates it in the anchor database. This way, when locating the page element again, multiple element location anchors can be used for more accurate and convenient location.

[0383] All generated virtual anchors are stored in an element anchor library, supporting efficient retrieval and reuse. For example, the anchor library index structure is as follows:

[0384] {

[0385] Element anchor library index;

[0386] Anchor ID index;

[0387] Used to quickly query anchor details based on ID;

[0388] Application - Element Mapping Index;

[0389] Search for all anchors of an application by application name;

[0390] Element semantic vector index (HNSW);

[0391] Used for semantic retrieval of similar elements;

[0392] Scene-element association index;

[0393] Query the anchor points of elements involved in a specific operation scenario;

[0394] Version-Policy Mapping Index;

[0395] Search for applicable location strategies by application version;

[0396] }

[0397] One example of anchor point retrieval is locating the search button when generating automated scenario tasks on an e-commerce website. When the server (system) needs to generate scenario tasks related to "product search" for the e-commerce website, it will search for the search button of the e-commerce website in the anchor point library.

[0398] Anchor point retrieval can also be exemplified by migrating the operational scenarios of e-commerce website 1 to e-commerce website 2. For instance, the system has already learned the operational scenarios of searching for products on e-commerce website 1, and now the user wants to perform a similar operation on e-commerce website 2. The system will search for scenario-matching operations on e-commerce website 1 and element positioning anchors (such as search buttons) on e-commerce website 2. If semantic vector similarity calculations determine that the semantic similarity between the search button on e-commerce website 1 and the search button on e-commerce website 2 reaches 0.92, it indicates that they are highly consistent in function. Therefore, the system can directly reuse the operational logic of the search clustering scenarios obtained from e-commerce website 1, and only needs to replace the element positioning anchors to perform the same search operation on e-commerce website 2.

[0399] In this application, the automated process refers to a DSL (Domain Specific Language). This application designs a unified DSL language to describe the automated operations of all applications and achieves general automated control by dynamically generating DSLs (i.e., scenario tasks) through LLM. The syntax structure of the DSL in this application is abstracted as: Resource (Target) + Action + Parameters + Preconditions / Postconditions + Rollback Branches, supporting structured modeling and executable expression of task plans.

[0400] The design goals of the DSL are: universality (supporting all operation types of browsers and native applications), readability (the DSL should be close to natural language, easy to understand and debug), verifiability (including preconditions / postconditions, supporting automatic verification), security (including rollback mechanisms and exception flags), and extensibility (supporting custom operation types and parameters).

[0401] The DSL consists of the following core elements, forming a complete automated execution plan:

[0402] {

[0403] plan_id: A unique identifier for the plan;

[0404] plan_name: Plan name;

[0405] description: Natural language description;

[0406] scene_id: The ID of the associated operation scene;

[0407] trigger_conditions: A list of trigger conditions;

[0408] trigger_type: Trigger type (time / event / data_change / user_command);

[0409] condition_expression: The conditional expression;

[0410] check_interval: Check frequency (for long-running tasks);

[0411] preconditions: A list of preconditions;

[0412] condition_type: Condition type (expression / element_exists / api_check); condition_value: Condition value;

[0413] error_message: The message displayed when the condition is not met;

[0414] steps: A list of execution steps;

[0415] step_id: Step ID;

[0416] step_description: Step description;

[0417] step_type: Step type (navigate / click / input / wait / api_call / script); target: Target resource;

[0418] target_type: Target type (url / element / api / system);

[0419] target_identifier: Target identifier (anchor ID / URL / API address);

[0420] target_params: Target parameters;

[0421] action: Operation definition;

[0422] action_type: Operation type;

[0423] action_params: Operation parameters;

[0424] variables: Variable definitions (variable parameters);

[0425] require_hitl: Whether manual confirmation is required;

[0426] on_error: Error handling strategy (abort / continue / retry / rollback);

[0427] branch: Branching condition (supports dynamic branching);

[0428] rollbacks: List of rollback steps; (Structure is the same as steps, but operation is reversed) verifications: List of post-verifications;

[0429] verification_name: Verification name;

[0430] verification_type: Verification type;

[0431] expected_result: Expected result;

[0432] actual_source: The actual source of the result;

[0433] risk_assessment: Anomaly assessment;

[0434] risk_level: Level of abnormality (low / medium / high / critical);

[0435] risk_factors: A list of abnormal factors;

[0436] mitigation_measures: mitigation measures;

[0437] metadata:

[0438] created_at: Creation time;

[0439] version: Version number;

[0440] execution_count: Number of executions;

[0441] success_rate: success rate;

[0442] }

[0443] Among them, Figure 8b In the process, the automated execution flow can be as follows: S821, the user initiates a task request using natural language commands; S822, determine the clustering scenarios related to the task request, which can be specified by the user or retrieved from a scenario library; S823, obtain the skeleton DSL (e.g., scenario matching operation as a mandatory step) from the clustering scenarios related to the task request; S824, generate variable operation parameters based on the task request using a large language model; S825, generate the scenario task based on the skeleton DSL and operation parameters (i.e., including the relevant steps of the scenario matching scenario to be executed and the operation parameters involved); S826, verify the legality of the scenario task; S827, after successful verification, infer and roll back the steps; S828, classify the content of the scenario task; S829, if the content classification result indicates that the scenario task is a normal task, then execute the scenario task; S8210, if verification fails, regenerate the operation parameters.

[0444] For high-frequency scenarios, the skeleton structure of the DSL is fixed, and the LLM only needs to generate variable parameters, significantly reducing generation latency. The skeleton DSL is defined as an abstract template for a specific scenario, containing: a fixed sequence of steps, fixed anchor points for positioning elements, and variable placeholders (filled by the LLM, i.e., operation parameters). Therefore, the structure and variables can be separated, greatly reducing the number of tokens generated by the LLM and the latency.

[0445] In this way, variable placeholders can be filled into a fixed sequence of steps to obtain the actual executable task steps, such as by filling in the large language model.

[0446] The DSL execution engine is responsible for parsing and executing the DSL plan, and includes complete error handling, rollback, and verification mechanisms. Figure 8cThe key steps include: (1) Precondition check: Before execution, verify whether the system status meets the execution conditions, such as whether a specific element exists on the page, by calling an external API and evaluating the expression, and whether the user has permission to access the page. (2) If the precondition check passes, perform anomaly assessment: Before execution, assess the anomaly level of the entire plan, and calculate the anomaly score based on factors such as whether it is an operation type (normal operation, or abnormal operation such as deletion / restart), target environment (online / test), and whether the operation supports rollback. If the score does not exceed the threshold, the operation can be executed automatically. If the score exceeds the threshold, HITL intervention is required, and multiple people need to confirm the operation if necessary. (3) After the anomaly assessment, the HITL manual confirmation mechanism will be triggered in the following situations: the precondition check fails, the overall anomaly score of the plan exceeds the threshold, the page element of a single step is a key element, the operation depends on external input (such as requiring a verification code), and the LLM inference cannot infer the next operation. If no manual confirmation is required, the step will continue to be executed.

[0447] The HITL confirmation page displays: operation description (describing the next step in natural language), operation goal (highlighting the corresponding operation element on the screen), error message, expected result, and rollback plan. User options include: approve (continue execution), reject (abstain execution and return an error), modify parameters (modify operation parameters and execute), and manual takeover (user manually operates, system learns).

[0448] (4) Element Location and Execution. Multiple location strategies are attempted based on priority, as described above. Upon successful location, the element is triggered to execute the steps. If location fails or the element is triggered but execution fails, an error is handled. If execution is successful, a check is performed to determine if subsequent steps are needed.

[0449] (5) Dynamic branching, supporting dynamic branching based on runtime state: branch types include conditional jump (skipping certain steps or jumping to a specified step based on a condition), loop (repeatedly executing certain steps until the condition is met), and exception branch (performing different processing based on the exception type).

[0450] For example, skipping the purchase of an item due to its being out of stock, here are some branch examples:

[0451] {

[0452] step_id: "S5";

[0453] branch:

[0454] condition: "product_price>max_price";

[0455] action: "skip_to_step(S10)" # Skip the purchase step;

[0456] condition: "product_stock==0"

[0457] Action: "abort_with_message(Product out of stock)"

[0458] condition: "else"

[0459] action: "continue" # Continue to the next step;

[0460] }

[0461] Among them, (6) Error handling and rollback: When a step fails, the corresponding operation is performed according to the error handling strategy. The error handling strategy may be: immediately stop without rolling back; retry the current step (the number of retries and intervals can be configured); execute the rollback step and restore the state before execution; record the error but continue to execute subsequent steps; request manual intervention when LLM inference fails.

[0462] (7) If there is no subsequent step, perform post-validation to verify whether the result meets expectations after execution. Validation types may include: element visibility validation (checking if a success message appears), text content validation (checking if the page contains the expected text), and API response validation (calling the API to check the data status). If the post-validation passes, rollback occurs; otherwise, logs are retrieved.

[0463] (8) Log recording: Completely records the execution process, supporting post-event analysis and problem tracing. Record content: Complete DSL of execution plan, execution time and results of each step, manual confirmation record (confirmer, confirmation time, confirmation result), element location details (location strategy used), error and exception information, rollback execution record, and screenshots of key operations.

[0464] In addition, a long-term task triggering mechanism can be configured. The triggering conditions for long-term tasks are continuously monitored, and the DSL plan is automatically executed when they are met. For example, triggering can be based on time, data changes (such as changes in product prices), events, page status, etc. No further limitations are specified here.

[0465] Among them, Figure 8d Long-term tasks can be configured, and detectors can be created at this time. Different long-term tasks have different triggering methods. For example, triggering by time can be done through a time scheduler, processing by data changes (such as changes in product prices) can be done through a data detector, triggering by events can be done through an event monitor, and triggering by page status can be done through a page detector, etc.

[0466] Specifically, the condition evaluation engine determines whether the execution conditions are met based on the aforementioned triggering methods. If the execution conditions are not met, the detection continues. If the execution conditions are met, DSL execution is triggered, meaning the execution engine performs the scenario task and notifies the user of the result, after which detection continues.

[0467] Therefore, this application is applicable to scenarios such as AI automation, browser automation, office assistance, and computer operation assistance, providing a universal cross-system automation control system suitable for scenarios requiring data acquisition, processing, and automated operation across multiple heterogeneous systems. For example, a single task: a scenario where an intelligent agent replaces or assists the user in operating the computer to complete a specific task. For example, a long-term task: where the user describes the timing of the task using natural language, and the system automatically executes the task after the detection conditions are met.

[0468] This application enables the following functionalities: operation sequence acquisition (constructing a unified heterogeneous dataset of user operation sequences through data acquisition, processing, and storage methods to achieve unified operation control across applications; the operation sequences include structured (e.g., click events, input content) and unstructured (e.g., screenshots, videos) multimodal information); operation scenario analysis (generating scenario descriptions based on key operations in the user operation sequence, and merging similar scenarios to generate automated operation processes); software element anchor point generation method (generating unique anchor points for key elements in applications / web pages through algorithms for locating and replaying operations; simultaneously generating verifiable DSL codes for different elements in the operation scenario for subsequent rapid location of corresponding elements); and DSL-driven automated control (controlling automated operations through a unified DSL; dynamically generating DSLs based on LLM during task execution to achieve universal automated control). This results in an accurate, low-latency, highly open / scalable, and self-evolving automatic control system. This system achieves high accuracy and low latency (through DSL-driven automation algorithms, it ensures high precision and low latency automation control while maintaining HITL; for operations dependent on external input or abnormalities, manual verification is required), high openness and scalability (it supports interactive analysis of user operation history, expanding the boundaries of problem handling), and self-evolution (it can understand user intent and decompose it into specific operation steps; it supports analysis and reasoning of corresponding operation scenarios based on user operation sequences and generates corresponding automated operation processes; it supports automated execution of operation processes based on user natural language commands and other triggering conditions; it supports real-time reasoning of the next operation based on the user's current operation, and the system continuously trains the model based on user feedback).

[0469] This application proposes an intelligent agent capable of automating processes. This agent is an autonomous module with capabilities in perception (data access), memory (context caching), reasoning (policy / model), and action (invoking tools or executing scripts). This intelligent agent (system) expands the boundaries of machine problem-solving capabilities, enabling AI to handle problems described in natural language. Instead of defining them through code, it can analyze user intent using a large language model and adaptively generate execution paths by combining tool and memory capabilities, such as identifying clustering scenarios and generating scenario tasks. Therefore, it can improve operational efficiency, reduce operational anomalies (by pre-checking and automatic rollback, reducing the error rate), lower the automation threshold (users can create automated tasks using natural language, without programming skills), support continuous learning (the system learns from historical execution records and continuously optimizes task execution strategies), and enhance compliance capabilities (providing a complete cross-application operation chain to meet compliance requirements).

[0470] Among them, Figure 8e This system consists of an interaction layer, a scheduling layer, an agent layer, an engine layer, and a storage layer. The interaction layer includes: historical operation query, natural language input, API interface, and visual interface; the scheduling layer includes: trigger management, long-term task detector, condition evaluation engine, and single-task scheduler; the agent layer includes: task decomposer, scene analysis agent, DSL generation agent, and execution coordination agent; the engine layer includes: operation acquisition engine, scene recognition engine, anchor point generation engine, and DSL execution engine; and the storage layer includes: operation sequence dataset, scene library, template library, and anchor point library.

[0471] This system has three core working modes. Mode 1: Operation Learning Mode, which can: (1) continuously record user operations by the operation acquisition engine; (2) store operation sequences in the Lance dataset; (3) analyze operation sequences by the scene recognition engine; (4) identify operation scenes; (5) generate scene descriptions and key steps by the scene analysis agent; (6) generate virtual anchors for key elements by the anchor point generation engine; (7) accumulate knowledge into the scene library and anchor point library.

[0472] Mode 2: Single task execution mode, which can: (1) The user inputs task requirements through natural language; (2) The task decomposer decomposes the requirements into sub-tasks; (3) The scenario analysis agent retrieves similar historical scenarios; (4) The DSL generation agent generates a DSL execution plan based on the scenario and anchor points; (5) The DSL execution engine drives cross-application automated operations; (6) The execution results are fed back to the user.

[0473] Mode 3: Long-term task detection mode, which can: (1) allow users to configure trigger conditions (time / data changes / page status, etc.); (2) allow the long-term task detector to continuously evaluate the trigger conditions; (3) automatically switch to single task execution mode when the conditions are met; (4) notify the user after execution and continue detection.

[0474] In this embodiment, the operation sequence dataset can be divided into multiple operation sequence groups using a sliding window. Each operation sequence group may represent a business operation step within a specific business scenario. Multiple operation sequence groups can then be clustered to obtain clusters. Each cluster can be considered a potential business scenario (clustering scenario, such as a search scenario). Therefore, the scenario matching operation (common step) within the corresponding clustering scenario can be determined from the operation sequences included in the cluster, thus obtaining the execution flow of the business scenario. Furthermore, the scenario matching operation and element positioning anchor points can be stored for subsequent automation process implementation. For example, when obtaining scenario matching operations from the scenario library to generate a clustering scenario task, the page element can be located using the element positioning anchor point to execute the scenario matching operation through the page element, thereby achieving automation for the clustering scenario. Therefore, by collecting and identifying the operation sequences of business operations, multiple business scenarios and the scenario matching operation within each business scenario can be quickly determined. This allows for process judgment and configuration of business scenarios during the user's actual operation, thereby optimizing the definition steps of the business process, improving definition efficiency and flexibility, and facilitating more convenient business process automation.

[0475] Please see Figure 9 , Figure 9 This is a schematic diagram of a data processing apparatus provided in an embodiment of this application. It should be noted that... Figure 9 The data processing apparatus shown is used to execute this application. Figure 3 , Figure 6 The methods in the illustrated embodiments are shown only in the parts relevant to the embodiments of this application for ease of explanation; specific technical details are not disclosed. Reference to this application is required. Figure 3 , Figure 6 The example shown. Figure 9 The example data processing device 1000 may include: a sequence processing module 1001 and an operation processing module 1002. Wherein:

[0476] The sequence processing module 1001 is used to obtain multiple operation sequence groups to be clustered; multiple operation sequence groups refer to those obtained by dividing the operation sequence dataset into windows through a sliding window.

[0477] The sequence processing module 1001 is also used to perform sequence clustering on multiple operation sequence groups based on the operation sequences in multiple operation sequence groups to obtain clusters related to the multiple operation sequence groups;

[0478] The operation processing module 1002 is used to determine the scene matching operation that matches the clustering scene corresponding to the clustering cluster based on the business operation represented by the operation sequence contained in the clustering cluster.

[0479] The operation processing module 1002 is also used to generate element positioning anchors for locating page elements for scene matching operations, so that when a scene task related to the clustering scene is obtained, the page elements are located in the task page of the scene task through the element positioning anchors, and the scene matching operation of the scene task is performed through the page elements.

[0480] The sequence processing module 1001 is also used for:

[0481] When a business operation of a preset operation type is detected on the current business page, the operation data of the business operation is obtained, and the operation sequence corresponding to the business operation is constructed based on the operation data;

[0482] The operation data includes at least one of the following: basic operation information, page information of the current business page, element information of the page element corresponding to the business operation, operation parameter information, and multimodal operation information.

[0483] The conditions for obtaining multimodal operation information include at least one of the following: the business operation is a critical operation; the number of business operations between the business operation and historical business operations reaches a preset number; historical business operations refer to business operations whose previously collected operation data contained multimodal operation information.

[0484] Key operations include at least one of the following: business operations of type 'end operation', business operations of type 'abnormal operation', the first business operation when the current business page and the business page displayed before the current business page do not belong to the same business process, the first business operation after the page stay time on the current business page reaches the preset time, and business operations when there is an error message on the current business page.

[0485] Multimodal operation information includes at least one of the following: a screenshot of the current business page, text information extracted from the screenshot, and visual features extracted from the screenshot.

[0486] The window length of the sliding window is determined based on a preset time, or based on the frequency of business operations indicated by the operation sequence in the operation sequence dataset, or based on the operation boundary indicated by the operation sequence in the operation sequence dataset.

[0487] The operation sequence dataset includes a first operation sequence and a second operation sequence. The operation boundary includes at least one of the following: the operation application indicated by the first operation sequence is different from the operation application indicated by the second operation sequence; the time interval between the operation time indicated by the first operation sequence and the operation time indicated by the second operation sequence reaches a preset interval; the operation type indicated by the second operation sequence is an end operation type; and the page domain name of the business page indicated by the first operation sequence is different from the page domain name of the business page indicated by the second operation sequence.

[0488] Specifically, when the sequence processing module 1001 performs sequence clustering on multiple operation sequence groups based on operation sequences in multiple operation sequence groups to obtain clusters related to the multiple operation sequence groups, it is used for:

[0489] Determine the sequence feature vector of the operation sequence in each operation sequence group to obtain the sequence feature vector of each operation sequence group;

[0490] Multiple business operation groups are clustered based on their sequence feature vectors to obtain at least one cluster; a cluster includes at least one operation sequence group.

[0491] Wherein, at least one cluster includes a target cluster, and the target cluster includes at least a target operation sequence group;

[0492] When the operation processing module 1002 determines the scene matching operation that matches the clustering scene corresponding to the cluster based on the business operations represented by the operation sequence contained in the cluster, it is specifically used for:

[0493] Retrieve the operation description text of the business operation indicated by the operation sequence in the target operation sequence group;

[0494] Scenario prompt text is generated based on the operation description text of the business operation indicated by the target operation sequence group and / or at least part of the operation sequence in the target operation sequence group; the scenario prompt text is used at least to determine the scenario matching operation for the clustering scenario;

[0495] The scene prompt text is input into the target language model, which then extracts the key operation description text from the operation description text of the business operation indicated by the target operation sequence group based on the scene prompt text. The key operation description text is the operation description text of the business operation that needs to be performed in the clustering scenario.

[0496] The business operations corresponding to the key operation description text are identified as scenario-matching operations.

[0497] The operation processing module 1002 is also used for:

[0498] Determine the scene confidence level for scene matching operations;

[0499] If the scene confidence is greater than or equal to the preset scene confidence, the scene matching operation is added to the scene library; the scene library is used to execute scene tasks.

[0500] If the scene confidence level is lower than the preset scene confidence level, the scene matching operation will be added to the scene library or the candidate library after user confirmation.

[0501] Specifically, the operation processing module 1002, when determining the scene confidence level for scene matching operations, is used for:

[0502] Determine the feature similarity between the sequence feature vectors of each operation sequence group in the target cluster, and determine the first confidence score based on the feature similarity; and / or,

[0503] Determine the frequency of occurrence of the scenario matching operation within the business operations represented by each operation sequence group, and determine the second confidence level based on the frequency of occurrence; and / or,

[0504] When the scene matching operation is determined by the target model based on the operation sequence in the target operation sequence group, the model confidence output by the target model for the scene matching operation is determined as the third confidence.

[0505] The scenario confidence level is determined based on at least one of the first confidence level, the second confidence level, and the third confidence level.

[0506] The operation processing module 1002 is also used for:

[0507] If a scenario matching operation is added to the candidate library, when a user performs a business operation related to a clustering scenario, the user's next business operation in the clustering scenario is predicted based on the scenario matching operation and the user's historical business operations; the predicted next business operation is the operation in the scenario matching operation.

[0508] If the business operation currently performed by the user is the same as the predicted next business operation, the scenario confidence of the scenario matching operation will be increased until the adjusted scenario confidence is greater than or equal to the preset scenario confidence.

[0509] If the business operation currently being performed by the user is different from the predicted next business operation, then the scenario confidence of the scenario matching operation will be reduced.

[0510] The operation processing module 1002 is also used for:

[0511] If a scenario matching operation is added to the scenario library, when a user performs a business operation related to a clustering scenario, the user's next business operation in the clustering scenario is predicted based on the scenario matching operation and the user's historical business operations; the predicted next business operation is the operation in the scenario matching operation.

[0512] If the business operation currently being performed by the user is different from the predicted next business operation, then the scenario matching operation will be adjusted.

[0513] Specifically, when generating element positioning anchor points for locating page elements, the operation processing module 1002 is used for:

[0514] The operation sequence based on scene matching operations generates positioning information for locating page elements, and generates positioning strategies for the positioning information. The positioning information and positioning strategies are used as anchor points for element positioning. The positioning strategy is used to indicate the positioning priority of multiple positioning information.

[0515] The positioning strategy includes at least one of the following: state features, element features, and element description text; the state features are obtained through the page information of the business page where the scene matching operation is located, and the element features and element description text are obtained through the element information of the page elements of the scene matching operation.

[0516] The operation processing module 1002 is also used for:

[0517] When a user performs a business operation related to a clustering scenario, the system predicts the user's next business operation in the clustering scenario based on the scenario matching operation and the user's historical business operations. The predicted next business operation is the operation in the scenario matching operation.

[0518] Generate operation prompts for the predicted next business operation and display the operation prompts on the business page of the clustering scenario;

[0519] After the user confirms the operation prompt, the anchor point of the predicted next business operation is obtained. The page element of the predicted next business operation is located on the business page through the anchor point, so as to execute the predicted next business operation through the located page element.

[0520] The scene matching operation is stored in the scene library, and the element positioning anchor point is stored in the anchor point library.

[0521] When the operation processing module 1002 is used to locate page elements in the task page of the scene task by using element positioning anchors when a scene task related to the clustering scene is obtained, it is specifically used for:

[0522] When a user submits a task requirement, the clustering scenario associated with the task requirement is determined, and the scenario matching operation for the clustering scenario is obtained from the scenario library.

[0523] Generate scene tasks related to the clustering scene based on scene matching operations and task requirements;

[0524] When performing a scene matching operation on the task page according to the scene task, the element positioning anchor is obtained from the anchor point library, and the page element is located on the task page through the element positioning anchor.

[0525] Among them, the element positioning anchor point includes multiple positioning information and positioning strategy used to position page elements. The positioning strategy is used to indicate the positioning priority of multiple positioning information.

[0526] When the operation processing module 1002 is used to locate page elements in the task page by using element positioning anchors, it is specifically used for:

[0527] Based on location priority, locate page elements in the task page according to multiple location information;

[0528] If a page element is located using the target location information from multiple location information sources, then adjust the location priority of the target location information.

[0529] If the page element cannot be located using multiple location information methods, an element confirmation message is generated so that the user can confirm the page element on the task page. The confirmed page element can then be used to update the element location anchor point.

[0530] Specifically, when the operation processing module 1002 performs scene matching operations for scene tasks via page elements, it is used for:

[0531] If the anchor point of an element indicates that the page element is a key element, a trigger message will be generated;

[0532] After the user confirms the trigger prompt, the page element is triggered to perform the scene matching operation.

[0533] The specific implementation methods of the sequence processing module and the operation processing module can be found in the description of the above embodiments, and will not be repeated here. It should be understood that the beneficial effects obtained by using the same method will also not be repeated here.

[0534] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 10As shown, the electronic device 2600 includes at least one processor 2601 and a memory 2602. Optionally, the electronic device may also include a network interface. The processor 2601, memory 2602, and network interface can exchange data. The network interface, controlled by the processor 2601, is used to send and receive messages. The memory 2602 stores computer programs, including program instructions. The processor 2601 executes the program instructions stored in the memory 2602. The processor 2601 is configured to invoke the program instructions to execute the aforementioned method. The memory 2602 may include volatile memory, such as random-access memory (RAM); the memory 2602 may also include non-volatile memory, such as flash memory, solid-state drive (SSD), etc.; the memory 2602 may also include combinations of the above types of memory.

[0535] Processor 2601 may be a central processing unit (CPU). In one embodiment, processor 2601 may also be a graphics processing unit (GPU). Processor 2601 may also be a combination of a CPU and a GPU. Processor 2601 may be used to invoke device control applications stored in memory 2602 to perform the above-described tasks. Figure 3 , Figure 6 The description of the data processing method in the corresponding embodiments can also be executed as described above. Figure 9 The description of the data processing apparatus in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated.

[0536] In specific implementations, the devices, processors, memory, etc., described in the embodiments of this application can execute the implementation methods described in the above method embodiments, or they can execute the implementation methods described in the embodiments of this application, which will not be repeated here.

[0537] This application also provides a computer-readable storage medium storing a computer program. The computer program includes program instructions, which, when executed by a processor, enable the processor to perform some or all of the steps described in the above method embodiments. Optionally, the computer storage medium can be volatile or non-volatile. The computer-readable storage medium may primarily include a program storage area and a data storage area. The program storage area may store an operating system, at least one application program required for a given function, etc.; the data storage area may store data created based on the use of blockchain nodes, etc.

[0538] This application provides a computer program product, which may include a computer program. When the computer program is executed by a processor, it can implement some or all of the steps in the above method, which will not be elaborated here.

[0539] In this article, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0540] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer storage medium, which can be a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0541] The above-disclosed embodiments are merely some of the embodiments of this application, and should not be construed as limiting the scope of this application. Those skilled in the art can understand that all or part of the processes for implementing the above embodiments, and equivalent changes made in accordance with the claims of this application, still fall within the scope of this application.

Claims

1. A data processing method, characterized in that, The method includes: Obtain multiple operation sequence groups to be clustered; the multiple operation sequence groups refer to those obtained by dividing the operation sequence dataset into windows using a sliding window. Based on the operation sequences in the plurality of operation sequence groups, sequence clustering is performed on the plurality of operation sequence groups to obtain clusters related to the plurality of operation sequence groups; Based on the business operations represented by the operation sequences contained in the cluster, determine the scene matching operation that matches the clustering scene corresponding to the cluster; Generate element positioning anchors for locating page elements in the scene matching operation. When a scene task related to the clustering scene is obtained, the element positioning anchors are used to locate the page element in the task page of the scene task, and the scene matching operation of the scene task is performed through the page element.

2. The method according to claim 1, characterized in that, The method further includes: When a business operation of a preset operation type is detected on the current business page, the operation data of the business operation is obtained, and an operation sequence corresponding to the business operation is constructed based on the operation data; The operation data includes at least one of the following: basic operation information, page information of the current business page, element information of the page element corresponding to the business operation, operation parameter information, and multimodal operation information.

3. The method according to claim 2, characterized in that, The conditions for obtaining the multimodal operation information include at least one of the following: the business operation is a critical operation; the number of business operations between the business operation and historical business operations reaches a preset number; the historical business operation refers to the business operation whose previously collected operation data contained multimodal operation information. The key operations include at least one of the following: a business operation of type 'end operation', a business operation of type 'abnormal operation', the first business operation when the current business page and the business pages displayed before the current business page do not belong to the same business process, the first business operation after the page dwell time on the current business page reaches a preset time, and a business operation when there is an error message on the current business page. The multimodal operation information includes at least one of the following: a screenshot of the current business page, text information extracted from the screenshot, and visual features extracted from the screenshot.

4. The method according to claim 1, characterized in that, The length of the sliding window is determined based on a preset time, or based on the frequency of business operations indicated by the operation sequences in the operation sequence dataset, or based on the operation boundaries indicated by the operation sequences in the operation sequence dataset. The operation sequence dataset includes a first operation sequence and a second operation sequence, and the operation boundary includes at least one of the following: the operation application indicated by the first operation sequence is different from the operation application indicated by the second operation sequence; the time interval between the operation time indicated by the first operation sequence and the operation time indicated by the second operation sequence reaches a preset interval; the operation type indicated by the second operation sequence is an end operation type; and the page domain name of the business page indicated by the first operation sequence is different from the page domain name of the business page indicated by the second operation sequence.

5. The method according to claim 1, characterized in that, The step of performing sequence clustering on the multiple operation sequence groups based on the operation sequences in the multiple operation sequence groups to obtain clusters related to the multiple operation sequence groups includes: Determine the sequence feature vector of the operation sequence in each operation sequence group to obtain the sequence feature vector of each operation sequence group; Based on the sequence feature vectors of the multiple operation sequence groups, the multiple business operation groups are clustered to obtain at least one cluster; a cluster includes at least one operation sequence group.

6. The method according to claim 5, characterized in that, The at least one cluster includes a target cluster, and the target cluster includes at least a target operation sequence group; The step of determining the scene matching operation that matches the clustering scene corresponding to the cluster based on the business operation represented by the operation sequence contained in the cluster includes: Obtain the operation description text of the business operation indicated by the operation sequence in the target operation sequence group; A scene prompt text is generated based on the operation description text of the business operation indicated by the target operation sequence group and / or at least a portion of the operation sequence in the target operation sequence group; the scene prompt text is used at least to determine the scene matching operation of the clustered scene; The scenario prompt text is input into the target language model, which then extracts key operation description text from the operation description text of the business operations indicated by the target operation sequence group based on the scenario prompt text. The key operation description text is the operation description text of the business operations that need to be performed in the clustering scenario. The business operation corresponding to the key operation description text is identified as the scenario matching operation.

7. The method according to claim 1, characterized in that, The method further includes: Determine the scene confidence level of the scene matching operation; If the scene confidence level is greater than or equal to the preset scene confidence level, the scene matching operation is added to the scene library; the scene library is used to execute the scene task. If the scene confidence level is less than the preset scene confidence level, the scene matching operation will be added to the scene library or the candidate library after user confirmation.

8. The method according to claim 7, characterized in that, Determining the scene confidence level of the scene matching operation includes: Determine the feature similarity between the sequence feature vectors of each operation sequence group in the target cluster, and determine a first confidence level based on the feature similarity; and / or, Determine the frequency of occurrence of the scenario matching operation in the business operations represented by each operation sequence group, and determine a second confidence level based on the frequency of occurrence; and / or, When the scene matching operation is determined by the target model based on the operation sequence in the target operation sequence group, the model confidence score output by the target model for the scene matching operation is determined as the third confidence score. The scenario confidence level is determined based on at least one of the first confidence level, the second confidence level, and the third confidence level.

9. The method according to claim 7, characterized in that, The method further includes: If the scenario matching operation is added to the candidate library, then when a user performs a business operation related to the clustering scenario, the user's next business operation in the clustering scenario is predicted based on the scenario matching operation and the user's historical business operations; the predicted next business operation is the operation in the scenario matching operation. If the user's current business operation is the same as the predicted next business operation, then the scenario confidence of the scenario matching operation is increased until the adjusted scenario confidence is greater than or equal to the preset scenario confidence. If the business operation currently being performed by the user is different from the predicted next business operation, then the scenario confidence of the scenario matching operation is reduced.

10. The method according to claim 7, characterized in that, The method further includes: If the scenario matching operation is added to the scenario library, when a user performs a business operation related to the clustering scenario, the user's next business operation in the clustering scenario is predicted based on the scenario matching operation and the user's historical business operations; the predicted next business operation is the operation in the scenario matching operation. If the user's current business operation is different from the predicted next business operation, then the scenario matching operation is adjusted.

11. The method according to claim 1, characterized in that, The generation of element positioning anchors for positioning the page elements includes: Based on the operation sequence of the scene matching operation, positioning information for locating the page element is generated, and a positioning strategy is generated for the positioning information. The positioning information and the positioning strategy are used as the positioning anchor point of the element. The positioning strategy is used to indicate the positioning priority of the multiple positioning information. The positioning strategy includes at least one of the following: state features, element features, and element description text; the state features are obtained through the page information of the business page where the scene matching operation is located, and the element features and the element description text are obtained through the element information of the page elements of the scene matching operation.

12. The method according to claim 1, characterized in that, The method further includes: When a user performs a business operation related to the clustering scenario, the user's next business operation in the clustering scenario is predicted based on the scenario matching operation and the user's historical business operations; the predicted next business operation is the operation in the scenario matching operation. Generate operation prompts for the predicted next business operation and display the operation prompts on the business page of the clustering scenario; After the user confirms the operation prompt information, the element positioning anchor point of the predicted next business operation is obtained. The page element of the predicted next business operation is located on the business page through the element positioning anchor point, so as to execute the predicted next business operation through the located page element.

13. The method according to claim 1, characterized in that, The scene matching operation is stored in the scene library, and the element positioning anchor point is stored in the anchor point library; When a scene task related to the clustering scene is obtained, locating the page element in the task page of the scene task using the element positioning anchor point includes: When a user submits a task requirement, the clustering scenario associated with the task requirement is determined, and the scenario matching operation for the clustering scenario is obtained from the scenario library. Based on the scene matching operation and the task requirements, generate scene tasks related to the clustering scene; When the scene matching operation is performed on the task page according to the scene task, the element positioning anchor is obtained from the anchor point library, and the page element is located on the task page through the element positioning anchor.

14. The method according to claim 13, characterized in that, The element positioning anchor point includes multiple positioning information and a positioning strategy for positioning the page element, and the positioning strategy is used to indicate the positioning priority of the multiple positioning information. The step of locating the page element in the task page using the element positioning anchor point includes: According to the positioning priority, the page elements are located in the task page sequentially based on the multiple positioning information; If the page element is located using the target location information among the multiple location information, then the location priority of the target location information is adjusted. If the page element cannot be located using the multiple location information, an element confirmation message is generated so that the user can confirm the page element on the task page using the element confirmation message; the confirmed page element can be used to update the element location anchor point.

15. The method according to claim 1, characterized in that, The scene matching operation performed through the page element includes: If the element's positioning anchor point indicates that the page element is a key element, then a trigger prompt message is generated; After the user confirms the trigger prompt, the page element is triggered to perform the scene matching operation.

16. A data processing apparatus, characterized in that, The device includes: The sequence processing module is used to obtain multiple operation sequence groups to be clustered; the multiple operation sequence groups refer to those obtained by dividing the operation sequence dataset into windows through a sliding window. The sequence processing module is further configured to perform sequence clustering on the plurality of operation sequence groups based on the operation sequences in the plurality of operation sequence groups, to obtain clusters related to the plurality of operation sequence groups; An operation processing module is used to determine a scene matching operation that matches the clustering scene corresponding to the clustering cluster based on the business operations represented by the operation sequence contained in the clustering cluster. The operation processing module is further configured to generate element positioning anchors for locating page elements in the scene matching operation, so that when a scene task related to the clustering scene is obtained, the page element is located in the task page of the scene task through the element positioning anchors, and the scene matching operation of the scene task is performed through the page element.

17. An electronic device, characterized in that, The system includes a processor and a memory, wherein the memory is used to store a computer program, the computer program including program instructions, and the processor is configured to invoke the program instructions to perform the method as described in any one of claims 1-15.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1-15.

19. A computer program product, characterized in that, The method includes a computer program comprising program instructions that, when executed by a processor, implement the method according to any one of claims 1-15.

Images