Target data automatic acquisition method, device, storage medium and program product

Abstract

The specification provides a target information automatic acquisition method, device, storage medium and program product. The method comprises: acquiring task information of a target information acquisition task, and acquiring current page state information of a target application to which the task information is directed; performing page operation decision based on the current page state information and the task information through a decision model constructed based on a language model, generating a page operation instruction and expected page state information thereof; executing the page operation instruction, reacquiring the page state information after execution, and determining whether the reacquired page state information conforms to the expected page state information; if yes, returning to the step of executing the page operation decision based on the reacquired page state information until reaching a target information acquisition page or completing the target information acquisition task; and if not, generating a page correction instruction according to an actual state, an expected state and the task information through the decision model according to a preset strategy, and executing through an operation execution interface.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to a method for automatically acquiring target data, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the development of mobile internet services, an increasing number of data acquisition processes need to be completed within third-party applications. For example, when conducting online business, users may need to access applications from banks, payment tools, government services, and telecommunications operators to query, export, download, or submit relevant information. This type of information is typically distributed across different functional pages within various applications.

[0003] In related technologies, users typically access the target application and retrieve data independently by following illustrated instructions or screen recording tutorials, or receive guidance from customer service via voice or text. This method relies heavily on the user's understanding and memorization of the operation path. When the page hierarchy is deep, the entry point names are inconsistent, or the page layout changes, users are prone to not finding the entry point, accidentally entering other pages, or repeatedly switching between different pages to view instructions, resulting in a time-consuming and unstable data retrieval process.

[0004] Other related technologies use automated scripts to assist users in performing interface operations. For example, they configure fixed operation paths or click positions for specific application versions and execute clicks, swipes, and inputs in a preset order. This type of method can reduce manual operation when the page structure is stable, but it usually relies on preset page states. When the application version is updated, the device resolution is different, the entry point position is adjusted, pop-ups appear, there is network latency, or the user intervenes midway, the actual page state may be inconsistent with the preset state, causing subsequent operations to fail or leading to error pages. Summary of the Invention

[0005] In view of the above, one or more embodiments of this specification provide the following technical solutions: According to a first aspect of one or more embodiments of this specification, a method for automatically acquiring target data is provided, comprising: Obtain target data, obtain task information for the task, and obtain the current page status information of the target application to which the task information is applied; By using a decision model built on a language model, page operation decisions are made based on the current page state information and the task information, generating the next page operation instruction to be performed and its corresponding expected page state information. The page operation instructions are executed through the operation execution interface to drive the target application to perform page jumps or in-page operations from the current page; After executing the page operation instruction, the page state information of the target application is reacquired, and it is determined whether the reacquired page state information matches the expected page state information. If the conditions are met, the newly acquired page state information will be used as the new current page state information, and the process will return to the steps of executing the page operation decision until the target data acquisition page is reached or the target data acquisition task is completed. If the condition is not met, the decision model generates a page correction instruction based on the reacquired page state information, the expected page state information, and the task information, and executes the page correction instruction through the operation execution interface.

[0006] According to a second aspect of the embodiments of this specification, an electronic device is provided, comprising: processor; Memory used to store processor-executable instructions; Wherein, when the processor executes the executable instructions, it is used to implement the method described in the first aspect.

[0007] According to a third aspect of the embodiments of this specification, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps of the method described in the first aspect.

[0008] According to a fourth aspect of the embodiments of this specification, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0009] As can be seen from the above embodiments, this specification obtains task information by acquiring target data and the current page status information of the target application to which the task information is applied. Since the status information comes from the real-time page of the target application, the decision model is able to perceive the actual operating environment from the beginning, avoiding the process from failing directly due to inconsistencies between the initial interface and the preset when the startup page layout changes due to version updates, device differences, or operational activities.

[0010] The current page state information and task information are input into the decision model to generate page operation instructions and their corresponding expected page state information. The operation instructions and expected states are jointly output by the same model based on an understanding of the page semantics and task objectives, ensuring that each operation carries the state reference benchmark that should be achieved at that step. This output method provides a clear basis for automatic comparison after execution verification, eliminating the need to write additional verification conditions for each step or manually determine whether the operation was successful.

[0011] After executing the operation command, retrieve the page status information again and determine whether it meets expectations.

[0012] If the expected result is achieved, the re-acquired state is used as the new current page state, and the decision model is returned to continue generating the next operation instruction, forming a closed-loop iteration of "perception → decision → execution → verification". Each step is verified before proceeding to the next round, and the state deviation of the previous step will not accumulate and amplify. This allows for stable progress to the target data acquisition page during multi-step transitions, reducing the probability of overall task failure due to the accumulation of intermediate anomalies when the link is too long.

[0013] If the expected results are not met, the decision-making model generates and executes corrective instructions based on the re-acquired page state information, expected page state information, and task information. The generation of corrective instructions takes into account the actual state, the expected state, and the original task objective, enabling the task to adaptively return to the normal flow when encountering dynamic disturbances.

[0014] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0015] Figure 1 This is an exemplary embodiment of an architecture diagram of a target data automatic acquisition system.

[0016] Figure 2 This is an exemplary embodiment of the architecture diagram of a terminal device.

[0017] Figure 3 This is an exemplary embodiment of the architecture diagram of a data acquisition client.

[0018] Figure 4 This is a flowchart of an exemplary embodiment of a method for automatically acquiring target data.

[0019] Figure 5 This is a schematic diagram of the structure of a device provided in an exemplary embodiment. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0021] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with relevant laws, regulations and standards, and corresponding operation entry points shall be provided for users to choose to authorize or refuse.

[0022] In some data retrieval scenarios, users need to access banking, payment tools, government services, telecommunications operators, or other third-party applications to query, export, download, or submit relevant target data. Because different applications vary in page hierarchy, function entry points, control styles, login status, and data retrieval processes, and the same application may also present different page states due to version updates, active overlays, network loading status, or changes in device environment, how to stably complete target data retrieval in a dynamic page environment is a crucial issue to consider in the automated processing of target data.

[0023] Related solutions have employed methods such as image and text guidance, screen recording tutorials, human customer service assistance, pre-authorized lookup, and fixed script automation to help users obtain target information. However, image and text guidance and human customer service assistance rely on users' understanding and execution of page paths, which can be easily interrupted by changes in entry points, deep page hierarchies, or user errors. While pre-authorized lookup can reduce user operations, it has significant limitations in scenarios involving active user interaction, privacy data, and compliance boundaries. Fixed script automation typically relies on preset coordinates, control identifiers, or fixed page paths; when the application interface is redesigned, button positions are adjusted, pop-ups are inserted, or loading states change, the script is prone to failure. Therefore, relying solely on static paths or fixed coordinates is insufficient to adapt to the dynamic changes in the state of third-party application pages.

[0024] Based on the above scenarios and processing requirements, this specification provides an automatic target data acquisition scheme. This scheme does not simply execute page operations sequentially according to a preset script, but rather dynamically decides on page operations based on the current page state and the target data acquisition task. After executing a page operation, it re-acquires the page state to determine whether the page changes meet expectations, thereby supporting subsequent operations, path correction, or process recovery. In this way, the adaptability and stability of page navigation during target data acquisition can be improved without relying on hard-coded coordinates.

[0025] Figure 1An exemplary architecture for an automatic target data acquisition system is illustrated. The automatic target data acquisition system may include a terminal device 10, a server 20, and a communication network 30 connecting the terminal device 10 and the server 20. The terminal device 10 may be a mobile phone, tablet computer, personal computer, laptop computer, PDA, wearable device, or other electronic device capable of running applications and displaying application pages. The communication network 30 may include a wired network, wireless network, local area network, wide area network, the Internet, or a combination thereof; this specification does not limit the embodiments thereto.

[0026] Multiple applications can run on terminal device 10. In the interactive scenarios of the embodiments of this specification, please refer to the different functional roles they play. Figure 2 These applications may include a task initiating application 101, a target application 102, and a data acquisition client 103.

[0027] The task initiating application 101, in terms of its functional role, acts as the initiator of the target data acquisition task. It can be a business processing application or other applications that require users to submit target data. For example, a user can select to submit bank statements, payment records, or other target data on the business processing page of the task initiating application 101, thereby generating a target data acquisition task and transmitting the task information of the target data acquisition task to the data acquisition client 103.

[0028] Target application 102, in terms of its functional role as the provider of target data, can provide data query, export, download or submission functions for banking applications, payment applications, government service applications, operator applications or other applications.

[0029] The data acquisition client 103 functions as an automated operation agent, responsible for acquiring task information for the target data acquisition task and, after obtaining user authorization, performing subsequent page operations on the target application 102. In terms of implementation, the data acquisition client 103 can be a standalone application, a mini-program, a quick app, a web page, a browser plugin, or a functional module embedded in other applications. The task initiating application 101 and the target application 102 can also be standalone applications or other forms.

[0030] In one possible implementation, the target data acquisition task can also be initiated by the data acquisition client 103. The data acquisition client 103 can provide a human-computer dialogue interface, where users can input data acquisition requests in natural language or other input formats, such as "Send me the bills for June." The data acquisition client 103 can perform semantic parsing on the data acquisition request to determine task information such as the data type, time range, target application, acquisition method, or receiving method of the data to be acquired, and generate the target data acquisition task accordingly.

[0031] In another possible implementation, the user can also input a data acquisition request in natural language or other input forms through the human-computer dialogue interface provided by the large model application. The large model application can then transmit task information generated based on the data acquisition request to the data acquisition client 103, or the data acquisition client 103 can be implemented by a functional module within the large model application that has automation capabilities. Thus, the task initiation function and the data acquisition function can be implemented separately by different applications, or integrated into the same application or the same human-computer dialogue interface.

[0032] For example, the data acquisition client 103 can be an intelligent agent program running on the terminal device 10, used to assist in executing the target data acquisition task after obtaining user authorization. This intelligent agent program can invoke model capabilities deployed locally on the terminal device 10 and / or on the server 20 to gain the ability to understand the page content of the target application 102 and the ability to make decisions regarding page operation processes. The invoked model capabilities can include at least one of a large language model, a decision model 40 built based on a language model, a visual language model, a text recognition model, an interface element detection model, and a page intent recognition model. In one possible implementation, the data acquisition client 103 can directly invoke the large language model, which generates page operation decisions based on task information and page state information. In another possible implementation, a decision model 40 can be built on the large language model through prompt configuration, knowledge enhancement, parameter adjustment, or task adaptation, and the decision model 40 performs page understanding, path planning, and page operation decisions. For page state information containing screen images, a large language model or a visual language model with visual information processing capabilities can also be invoked for processing.

[0033] The language model can be either an LLM (Large Language Model) or a VLM (Vision-Language Model). An LLM is a large-scale neural network model with text as its primary input and output modality, while a VLM is a multimodal language model capable of processing both image and text information simultaneously. Depending on the actual deployment conditions, the decision model 40 can flexibly choose either an LLM or a VLM as its underlying implementation without changing the upper-level decision logic and system architecture.

[0034] In one implementation, the decision model 40 built based on the language model can be deployed locally on the terminal device 10. In another implementation, the decision model 40 built based on the language model can be deployed on the server 20, and the terminal device 10 interacts with the server 20 through the communication network 30. The server 20 can be a standalone physical server, a server cluster, a cloud server, or a virtual server, and this specification does not limit this.

[0035] Under the above system architecture, the terminal device 10 can serve as the main execution side of the automatic acquisition method of target data, and complete tasks such as task reception, page status acquisition, page operation execution and process recovery; the server 20 can provide decision models 40 based on language models, application description libraries, process template libraries, historical trajectory libraries or other auxiliary decision-making resources when needed.

[0036] When the data acquisition client 103 is implemented as a standalone application, it launches the target application 102 by calling the operating system interface and perceives and operates the interface of the target application 102, executing a cross-application automated process. When the data acquisition client 103 is implemented as a functional module embedded in other applications, it can share the same application process with the task initiating application 101 or the target application 102, and communicate and schedule through internal interfaces.

[0037] Figure 3 It shows Figure 2 An exemplary architecture of a data acquisition client 103 is provided. The data acquisition client 103 can be internally divided into multiple functional modules, which work collaboratively to achieve a complete automated closed loop from page awareness to operation execution and exception recovery. In one embodiment of this specification, the data acquisition client 103 may include a screen understanding module 1031, an intent decision module 1032, an execution control module 1033, and a user interaction module 1034.

[0038] The screen understanding module 1031 is responsible for digitally perceiving and analyzing the current interface of the target application 102.

[0039] This module can call a text recognition model to perform OCR (Optical Character Recognition) processing on the application interface image captured by the terminal device 10 screen, and extract the visible text content on the page.

[0040] This module can also invoke the interface element detection model to identify all interactive user interface elements on the page, such as buttons, input boxes, list items, icons, etc., and determine the type, display position on the screen, interaction attributes, and associated text description of each interactive element. The interface element detection model can be a target detection model based on convolutional neural networks or a detection model based on image feature matching. The specific implementation method is not limited in the embodiments of this specification.

[0041] This module can also invoke the page intent recognition model to classify the current page's functional state based on the overall features of the screen image, the recognized text content, and the layout of interface elements. For example, it can determine whether the current page is a login page, application homepage, data list page, data export page, or submission confirmation page. By integrating this information, the screen understanding module 1031 generates a structured current page state information, providing a state description for the downstream intent decision module 1032.

[0042] The intent decision module 1032 generates specific operation instructions for the next step based on the current page state information and task objectives. This module interacts directly with the decision model 40, which is built based on a language model. The intent decision module 1032 inputs the current page state information generated by the screen understanding module 1031 and the task information of this task into the decision model 40. Based on its understanding and reasoning ability of complex tasks, the decision model 40 executes the page operation decision process, generating page operation instructions and expected page state information.

[0043] The execution control module 1033 is responsible for translating page operation instructions into actual operations on the target application 102 and obtaining visual feedback from the target application 102 after the operation. This module executes page operation instructions by calling the operating system-level interface of the terminal device 10.

[0044] For example, it can call the accessibility service interface of terminal device 10 to map the operation intent described in the page operation instructions into a specific sequence of actions that the accessibility service can understand and execute. This module can also utilize simulation operation technology to complete operations by simulating screen touch events or keyboard input events. After each operation instruction is executed, the execution control module 1033 triggers a screenshot feedback action, that is, takes a screenshot of the display interface of the current target application 102. This screenshot image will serve as the input source for the screen understanding module 1031 to perform the next perception task. Through such an atomic operation loop of "decision-execution-screenshot," the data acquisition client can progressively drive the state of the target application forward.

[0045] The user interaction module 1034 is responsible for handling boundary situations requiring user interaction during automated processes, particularly scenarios involving privacy, security, and confirmation of pre-task conditions. When the task progresses to a page requiring the user to enter a password, perform biometric authentication, fill in sensitive personal information, or confirm final data submission—a page requiring privacy-sensitive operations—the screen understanding module 1031 identifies the intent of such pages and notifies the execution control module 1033. The execution control module 1033 then pauses the subsequent sequence of automated operation instructions and the periodic screenshotting behavior of the screen understanding module 1031, transferring control to the user interaction module 1034. At this time, the user interaction module 1034 can generate and output prompts to the user to complete the current operation. These prompts can be floating windows overlaid on the screen, voice announcements, or notification bar messages.

[0046] While outputting the prompt message, this module can also render a continue control on the interface. After the user completes the necessary privacy operation and clicks the continue control, the user interaction module 1034 transmits a recovery signal to the execution control module 1033. The execution control module 1033 reactivates the screen understanding module 1031, re-perceives the page after the operation, and passes the new page state information to the intent decision module 1032, thereby enabling the automated process that was paused due to the privacy operation to resume seamlessly.

[0047] The data acquisition client 103 can connect to the cloud knowledge base 200 deployed on the server 20 via a network interface to provide macro-level task process guidance and domain knowledge for the intent decision module 1032.

[0048] The cloud-based knowledge base may include an application description library 201, which stores page description information for each relevant page of different applications for different types of data acquisition tasks, and assists the intent decision module 1032 in improving the accuracy of the path when planning the page navigation path for the first time.

[0049] The cloud-based knowledge base can also include a process template library 202, which stores common page navigation paths for different types of data retrieval tasks for reference.

[0050] The cloud-based knowledge base can also include a historical trajectory library 203, which stores the actual execution trajectory of past successful executions and the scenario information during execution. When encountering a special page that cannot be covered by the standard process, the intent decision module 1032 can query the historical trajectory library to find a successful trajectory that matches the current execution scenario and extract a reusable navigation path or recovery strategy from it.

[0051] The modular architecture described above decouples the responsibilities of perception, decision-making, execution, and interaction. The introduction of a cloud-based knowledge base enables the data acquisition client 103 to make decisions based on structured domain knowledge and historical experience when faced with target applications of different versions or operational interference, rather than being limited to pre-built fixed scripts.

[0052] Please see Figure 4 This diagram illustrates a flowchart of an automatic target data acquisition method, which can be executed by a data acquisition client in the aforementioned terminal device. After obtaining user authorization, this intelligent agent program can perceive and understand the current interface of the target application and, based on the target data acquisition task to be completed, autonomously drive the target application to perform page navigation or in-page interaction operations, thereby replacing the user in completing the tedious data search and acquisition process. The method includes: In S400, the task information for the target data acquisition task is obtained, and the current page state information of the target application to which the task information is applied is obtained.

[0053] The target data acquisition task defines the ultimate purpose of this automation process. Task information may include a basic description of this purpose; for example, it may indicate the type of data to be acquired, specific identifiers associated with the data, or constraints such as a time frame. After obtaining authorization, the data acquisition client first parses the task information to determine the target application to be operated on. The target application can be any application installed on the terminal device.

[0054] For example, the data acquisition client launches the target application by calling the operating system's application launch interface. Alternatively, the target application can be manually opened by the user, and then the data acquisition client takes over. Or, the data acquisition client can directly take over from the target application that is already running in the background. It can also start execution from an intermediate page of the target data acquisition task, without needing to enter from the application's launch homepage. This embodiment does not impose any limitations on this.

[0055] Once the target application enters an operational state, the data acquisition client immediately begins to perceive its initial state. The current page state information characterizes the content and operational status of the target application's currently displayed page, ensuring that subsequent page operation decisions are based on the actual page of the target application. Because the current page state information includes a real-time representation of the page after startup or takeover, the decision-making model receives state input synchronized with the current operating environment from the start of the task. Therefore, even if the target application's homepage layout changes due to version updates, device differences, or operational activities, subsequent decisions can still be based on the actually presented page content, avoiding task failure directly caused by inconsistencies between the initial page and the preset.

[0056] The process of obtaining current page state information may include capturing a screenshot of the terminal device screen to obtain a screen image, and then obtaining the current page state information by understanding the screen image. The page state information of any page may include at least one of the following: a screen image of the page; text recognition results obtained by performing text recognition on the screen image; interactive element information obtained by detecting interface elements on the screen image; page intent information obtained by page intent recognition based on the screen image, text recognition results, and / or interactive element information; and key element information relevant to the target data acquisition task, selected from the interactive element information.

[0057] A screen image is a pixel-level record of the target application's current display interface. Text recognition results can be obtained by processing the screen image using an OCR (Optical Character Recognition) model. An OCR model can detect text regions in an image and convert the characters within those regions into machine-readable text sequences. The text recognition results can include the text content of each text block, the coordinates of the text block's position on the screen, and the paragraph hierarchy between text blocks.

[0058] Information about interactive elements can be obtained by processing screen images using a UI element detection model. This model is a target detection model that can locate user interface controls such as buttons, input boxes, radio buttons, checkboxes, sliders, tabs, list items, and icons in a screen image, and output the element type, display position on the screen, interaction attributes, and associated element text for each interactive element. Interaction attributes indicate whether the element is clickable, allows input, can be scrolled, or can be long-pressed.

[0059] Page intent information reflects the current functional state or business scenario of the page. Page intent recognition can be performed by a page intent recognition model, which can classify the current page into intent categories such as login page, application homepage, account overview page, profile list page, profile export page, or submission confirmation page, but is not limited to these.

[0060] In one embodiment, the page intent recognition model can employ a Vision-Language Model (VLM). The VLM receives a screen image and associated text description as input, fuses visual and semantic features through a cross-modal attention mechanism, and outputs a classification result or semantic description of the overall page intent. In another embodiment, the page intent recognition model can be a lightweight classification model based on a convolutional neural network or Transformer architecture, taking the concatenated features of the screen image and text recognition results as input, and outputting a page intent label.

[0061] The page intent recognition model can be deployed locally on the terminal device or on a server. The terminal device sends the screen image or the image features after de-identification to the server and receives the intent recognition result returned by the server.

[0062] Key element information is a subset of all interactive elements that is highly relevant to the current target data retrieval task. For example, when the task objective is to export transaction records for a specific time range, key element information could include buttons or input boxes directly associated with operations such as "time selection," "transaction record query," and "export." The filtering process can be completed by semantically matching keywords in the task information with the element text and element type of interactive elements, or the decision model can implicitly focus attention during the generation of page operation instructions.

[0063] In an optional implementation, to improve the accuracy of page structure perception, the screen image can be preprocessed before page intent recognition. The preprocessing process may include cropping the screen image into multiple region images according to page layout features. In one example, the screen image can be divided into a top region, a main text content region, and a bottom region. The top region may contain the status bar and navigation bar, the main text content region may contain the main page information, and the bottom region may contain a tab bar or operation bar. Corresponding region role identifiers can be assigned to different region images, and these region role identifiers, along with each region image, can be encoded into a structured state vector. The state vector can be further integrated with auxiliary configuration items such as text content, text paragraph structure, and temporary data from the system clipboard in the text recognition results. This state vector can then be used as input to the page intent recognition model to improve intent recognition accuracy.

[0064] In an optional implementation, the current page state information also includes natural language description information to guide the decision model 40 in focusing its attention.

[0065] For example, the text content identified in the top area (such as the page title and navigation bar labels) and the text content identified in the bottom area (such as tab bar options) can be used as contextual background information, while the text content identified in the main content area, the list of interactive elements, and their element text can be used as core decision information. This information is then integrated with spatial layout guidelines to generate a natural language instruction. For instance, the semantics of this instruction could be expressed as: The top area of ​​the current page contains the title "My Account" and navigation buttons "Back" and "Settings"; the main content area is located below the top area and contains a form titled "Transaction History Inquiry," which includes a start date input box, an end date input box, and a query button; please focus primarily on the task-related interactive elements in the main content area and ignore the fixed navigation bars at the top and bottom.

[0066] In this way, by understanding the spatial relationship indications and task relevance cues in this natural language description, decision model 40 can concentrate its limited reasoning resources on local areas of the page that play a key role in advancing the task, thereby reducing the interference of information from non-critical areas such as brand logos, advertising banners, or fixed navigation bars on the decision-making process.

[0067] In S402, a decision model based on a language model is used to make page operation decisions based on the current page state information and task information, and to generate the next page operation instruction to be performed and its corresponding expected page state information.

[0068] In the embodiments described in this specification, the decision model can be built based on a language model. The term "language model" is used here as a broader concept, encompassing various specific implementations such as LLM (Large Language Model) and VLM (Vision-Language Model). LLM is a large-scale neural network model that uses text as its primary input and output modality. When using LLM as the underlying language model to implement a decision model, the current page state information, including text recognition results, interactive element information, and page intent information, can be input into the LLM for reasoning and decision-making in the form of natural language text descriptions.

[0069] VLM (Visual Language Model) is a multimodal language model capable of processing both image and text information simultaneously. When using VLM as the underlying language model to implement a decision model, screen images or multiple regions of a cropped screen image can be directly passed to VLM as visual input without additional text conversion steps. VLM can extract visual features from the screen image through its internal visual encoder and fuse these visual features with text features such as text recognition results and interactive element information through a cross-modal attention mechanism, thereby making page operation decisions directly based on the joint representation of visual and semantic information.

[0070] Regardless of whether LLM or VLM is used, the core function of the decision model remains the same: receiving current page state information and task information as input, and generating page operation instructions and their corresponding expected page state information. Defining the decision model as being built based on a language model allows the technical solutions in the embodiments of this specification to adapt to different underlying model architectures. It enables flexible selection of LLM or VLM deployment based on the computing power, network environment, latency requirements, and privacy protection needs of the terminal device, without altering the upper-layer decision logic and system architecture.

[0071] Page action decision-making refers to the model's planning of a single action to be performed at the current moment to achieve the task objective, based on an understanding of the current page and the final goal. The output of this decision consists of two parts: first, the page action instruction, which specifies the concrete action to be performed next, such as clicking a specific button, entering specific text in an input box, or scrolling through a page list; second, the expected page state information, which is the decision model's prediction of the page state that the target application should present after executing the action instruction. This method of jointly generating "actions" and "expected consequences of actions" provides a clear and automatically comparable benchmark for subsequent automated verification, enabling flexible responses to dynamic changes in page layout.

[0072] For example, the expected page state information includes at least one of the following: expected page intent, expected page keywords, expected interactive elements, expected target key elements, and expected page redirection result.

[0073] 1. Page operation decisions based on knowledge base.

[0074] In one possible implementation, when making page operation decisions, the decision model can leverage a pre-set knowledge base to compensate for the limitations of the general language model in terms of specific application page knowledge. When the decision model relies solely on current page state information and basic task information, it faces a technical limitation: while the general language model possesses strong capabilities in semantic understanding and reasoning, it typically lacks specific knowledge of the target application's internal page structure, entry features, functional module layout, and inter-page navigation relationships. This lack of application-level page knowledge makes it difficult for the decision model to plan a reliable and executable page navigation path from the current page in the early stages of task execution, resulting in significant uncertainty in the initial path planning.

[0075] To address the aforementioned issues, in this embodiment, the step of obtaining task information for a target data acquisition task may include: obtaining basic task information from the application that triggered the target data acquisition task; and obtaining a set of page description information for the target application related to the target data acquisition task, containing at least one page, from a pre-defined application description library. The basic task information describes the core objective of this data acquisition task, such as the type of data to be acquired, associated account identifiers, or time range constraints. The application description library is a pre-built structured knowledge set used to store page description information for each relevant page of different applications for different types of data acquisition tasks. The page description information may include at least one of the following: page identifier, page title, page structure information, control semantic information, page appearance feature information, function entry description information, page screenshot information, and page location auxiliary information for each page involved in the task.

[0076] In the process of page operation decision-making, the decision model's handling of page navigation paths can be divided into two scenarios.

[0077] The first scenario involves a page navigation path from the current page to the target data retrieval page that has not yet been established. This typically occurs when a task has just started, the process needs to be replanned after an abnormal interruption, or an existing path is deemed unusable. In this case, the decision model can receive basic task information, a set of page description information, and current page state information as joint inputs. Based on this information, the decision model understands the task-related pages of the target application and, combined with the actual location of the current page within the application, plans a page navigation path that gradually navigates from the current page to the target data retrieval page. This path can be an ordered sequence of multiple page nodes and jump operations between nodes.

[0078] After planning the page navigation path, the decision model can perform path reasoning based on the path and the current page state information to determine the specific operation actions from the current node to the next node, thereby generating the next page operation instructions and their corresponding expected page state information.

[0079] The second scenario is when a page navigation path has already been established. This typically occurs during the normal step-by-step phase of task execution. In this case, the decision model does not need to replan the overall path; instead, it can directly perform path reasoning based on the current page state information and the existing page navigation path.

[0080] It should be noted that the path reasoning step is technically necessary in both the first and second scenarios described above. This is because the page navigation path is planned and generated based on page description information in the application description library, which stores abstract descriptions of the target application's page structure, entry features, and navigation relationships in a standard or typical state.

[0081] In a real-world operating environment, the actual page of the target application may differ from the standard description due to factors such as version updates, differences in account permissions, operational activity configurations, or user history. For example, the standard description may show the text of a button as "Query Now," while the actual page may show the text as "Query Transaction Records." Similarly, the standard description may not show an advertising banner at the top of the page, while the actual page may show a closable promotional pop-up.

[0082] If the decision model mechanically generates operation instructions based on the preset node information and operation sequence in the page navigation path after it has been formed, without combining the current page state information for real-time reasoning, the aforementioned page changes may cause the generated page operation instructions to be mismatched with the interactive elements in the actual page, thereby causing the automated process to fail.

[0083] The purpose of path reasoning is to bridge the gap between static path descriptions and dynamic page states. During path reasoning, the decision model semantically compares and matches the current page state information with the expected page corresponding to the current node in the page navigation path. This matching does not require complete consistency, but rather the decision model understands the actual layout and content of the current page to identify the target interactive element semantically corresponding to the path node. For example, when a path node indicates that a button for initiating a query needs to be clicked, the decision model can find a button labeled "Query Transaction Records" in the list of interactive elements on the current page through semantic similarity matching, even if the standard description label for that button is "Query Now".

[0084] Based on this, the decision model can generate page operation instructions that closely match the actual state of the current page, such as precisely pointing the click target to the display position and interaction attributes of the matched button. Simultaneously, the decision model can dynamically update the expected page state information based on the actual characteristics of the current page, ensuring that the expected page state reflects the actual changes the current page may undergo after the operation, rather than relying solely on templated predictions in standard descriptions.

[0085] Through the aforementioned path reasoning mechanism, the decision model can maintain the overall stability of the path direction by utilizing the prior knowledge provided by the application description library, while flexibly adapting to the dynamic changes of the target application page through real-time perception and semantic matching. This allows it to generate accurate and executable operation instructions even when there are unstructured changes to the page, thereby improving the robustness and success rate of the automated data acquisition process in real-world operating environments.

[0086] In the above embodiments, when planning page navigation paths, the decision model can utilize an application description library to obtain a set of page description information for a specific task within the target application. This allows it to acquire application-level page topology, entry features, and descriptions of relevant pages. The application description library provides fine-grained page knowledge tightly bound to the target application, enabling the decision model to accurately understand which pages within the target application are relevant to the current task, what interactive elements these pages contain, and how to navigate between pages. This application-level page knowledge provides the necessary underlying information support for path planning.

[0087] In a further embodiment, to constrain and guide the path planning process at a higher level, the step of obtaining task information for the target data acquisition task may further include: retrieving a general page navigation path corresponding to the target data acquisition task from a preset process template library. The process template library is a structured knowledge base used to store general page navigation paths for different types of data acquisition tasks. Unlike application description libraries, which focus on describing the details of pages within a specific application, the process template library focuses on depicting the general sequence of steps for a certain type of data acquisition task at the abstract process level.

[0088] A generic page navigation path can be represented as an ordered sequence of multiple process stage nodes. Each process stage node corresponds to a page intent category with a clear functional semantics, such as "login authentication," "account overview," "function entry location," "condition settings," and "result confirmation," without needing to be bound to a specific page identifier in the target application. This abstract representation, decoupled from specific applications, allows the same generic page navigation path to be applied to the same data retrieval task in different target applications.

[0089] During page operation decision-making, when planning the page navigation path from the current page to the target information retrieval page, the decision model can utilize two knowledge sources simultaneously: an application description library and a process template library. The decision model can receive basic task information, a set of page description information obtained from the application description library, current page state information, and relevant general page navigation paths obtained from the process template library as joint inputs. The decision model can first parse the functional semantics of each process stage node in the general page navigation path, and then retrieve candidate pages from the page description information set that match the semantics of each process stage node.

[0090] For example, the "condition setting" stage node in the general path can be matched to a page in the target application that has time range selection or filtering condition setting functions. After establishing the semantic matching mapping, the decision model can adjust the general page navigation path based on the actual starting position indicated by the current page state information, resulting in a page navigation path that conforms to both the macro-process of the task and the specific page topology of the target application. The adjustment process may include: starting from the node corresponding to the current page state, connecting the candidate page links determined by the semantic matching mapping to form a specific page navigation path; when there are multiple candidate pages in the target application that can all satisfy the semantics of a certain process stage, the appropriate candidate page can be selected based on the actual state of the current page and the cost of jumping between pages.

[0091] By introducing a process template library, the decision model can simultaneously obtain two levels of knowledge support during path planning: application-level fine-grained page knowledge provided by the application description library, and task-level macro-process constraints provided by the process template library. The application description library enables path planning to be precisely applied to specific pages and controls within the target application, while the process template library provides a validated process skeleton, ensuring that the generated page navigation paths better align with the typical operational flow of this type of data retrieval task. The collaborative use of these two knowledge bases eliminates the need for disordered exploration across all relevant pages of the target application during path planning. Instead, it allows for targeted page matching and adaptation adjustments guided by the macro-process skeleton, thereby improving the efficiency of path planning and the degree of matching between the generated paths and the target data retrieval task at the process level. Furthermore, since the general page navigation path is decoupled from specific applications, the same template path can be reused for similar tasks in multiple different target applications, enhancing the reusability of the knowledge base.

[0092] In another possible implementation, based on providing application-level page knowledge to the decision model through an application description library, or further providing task-level macro-process constraints to the decision model through a process template library, this specification also provides another optional path determination method. In actual operating environments, the page state of the target application may exhibit diverse manifestations due to factors such as version updates, operational activity configurations, differences in account permissions, or device environments. For certain specific types of execution environments, there may be operation trajectories that have been successfully executed in the past, and these trajectories contain valid path information that has been verified under specific scenario conditions.

[0093] To leverage such historical experience to aid decision-making, in one embodiment, the step of obtaining task information for the target data acquisition task may further include: retrieving historical execution trajectories and their corresponding execution scenario information from a preset historical trajectory database. The historical trajectory database is a data set used to store historical execution trajectories and their corresponding execution scenario information for different types of data acquisition tasks. Each historical execution trajectory may be a record of the complete page operation sequence traversed by the method provided in the embodiments of this specification when successfully completing the same or similar types of data acquisition tasks in the past.

[0094] Historical execution traces can include page state records for each page traversed from the task initiation page to the target data acquisition page, the operation instructions executed on each page, and the navigation relationships between pages. Execution scenario information stored in association with the historical execution trace describes the characteristics of the runtime environment when the trace was generated. This information may include the target application version identifier, the terminal device's operating system information, the user account's permission type, and page layout feature identifiers.

[0095] During the execution of an automated process, when a page navigation path has not yet been formed, or a formed page navigation path cannot proceed to the target data retrieval page, the decision model can trigger a historical trajectory reuse mechanism. The trigger condition "a formed page navigation path cannot proceed" can include: multiple consecutive instances where the page state does not match expectations while executing along the current path; the expected page corresponding to a node in the path not existing or having been replaced; or the page entry point on which the path depends being temporarily closed due to operational configuration changes. In these situations, the decision model can attempt to retrieve historical execution trajectories from the historical trajectory database that are the same as or similar to the current target data retrieval task type.

[0096] If one or more candidate historical execution trajectories are successfully obtained, the decision model can first match the execution scenario information associated with the candidate historical execution trajectory with the current execution scenario indicated by the current page state information. The current execution scenario can be characterized by scenario features that can be extracted from the current page state information, such as the current version number of the target application, the permission identifier of the currently logged-in account, and the layout feature vector or page intent classification result of the current page. The matching process can calculate the feature similarity between the execution scenario information of the candidate historical trajectory and the current execution scenario. When the similarity reaches a preset condition, that is, when the execution scenarios are determined to match, the candidate historical execution trajectory can be regarded as a valid reference applicable to the current execution scenario.

[0097] After identifying a matching historical execution trajectory, the decision model can determine a page navigation path from the current page to the target data acquisition page based on the current page state information. The decision model can semantically compare the current page state information with the various page states recorded in the historical execution trajectory to locate the node position corresponding to the current page in the historical trajectory. After location, the decision model can extract the operation sequence from the current corresponding node to the target data acquisition page in the historical trajectory and use it as the newly determined page navigation path. Furthermore, after determining this path, the decision model can still adapt and adjust the path by combining the current page state information and page description information to ensure that the specific operation instructions in the path are consistent with the actual interactive elements on the current page.

[0098] By introducing a historical trajectory library and a reuse mechanism based on execution scenario matching, the embodiments in this specification can obtain additional path references from historical successes, in addition to the application description library and process template library. The conditional constraints of execution scenario matching ensure that the reused historical trajectories have scenario-level similarity to the current operating environment, thereby reducing the risk of reused trajectories becoming inapplicable due to version differences or environmental changes. When faced with diverse page states caused by version fragmentation, personalized configurations, or temporary operational activities, this method can utilize operational experience validated in similar scenarios to determine feasible page navigation paths, providing a supplementary path determination approach for the automated data acquisition process.

[0099] 2. Use the fine-tuned decision model to make page operation decisions.

[0100] In another possible implementation, the decision model built upon the language model can be a fine-tuned model. Fine-tuning is a transfer learning technique that updates the parameters of a pre-trained language model on task-specific data, thereby optimizing the model's performance on that task. Through fine-tuning, general semantic understanding capabilities can be transferred and focused on the specific scenario of page operation decisions, enabling the model to more consistently output page operation instructions and expected page state information that meet the task requirements.

[0101] The fine-tuning process can include three stages: operation trajectory acquisition, training sample construction, and model parameter update.

[0102] During the operation trajectory collection phase, operation trajectories for different data acquisition tasks from different applications can be collected. An operation trajectory is a record of the complete data acquisition process executed manually or semi-automatically within the target application, and can include operation information for each step from task initiation to data acquisition completion.

[0103] Each operation trajectory can include page state information for each page involved in the task execution, page operation behaviors performed in each page state, and page state change information resulting from the execution of the page operation behavior. The specific content of the page state information can be found in the description above. Page operation behaviors can record the operation type (e.g., click, input, swipe, return, etc.), operation target (e.g., the type and text label of the operated element), and operation parameters (e.g., input text content, swipe direction, etc.). Page state change information can describe the differences between the page before and after the operation, such as newly appearing interface elements, disappeared interface elements, and changes in page intent.

[0104] During the training sample construction phase, training samples for fine-tuning can be constructed based on the collected operation trajectories. Each training sample can be constructed by extracting data corresponding to one operation step from the operation trajectory. The input part of the training sample can include: page state information before the operation step is executed, and task information of the current task. The task information can provide a global task context. The supervision label part of the training sample can include: the page operation instruction corresponding to the operation step, and the expected page state information after executing the operation instruction. The page operation instruction specifies the operation that should be performed in the current page state, while the expected page state information provides a description of the state that the page should reach after the operation is executed.

[0105] During the model parameter update phase, the language model can be fine-tuned using constructed training samples. The goal of fine-tuning is to adjust the parameters of the language model so that, given the input page state information and task information, the model can accurately predict the corresponding page operation instructions and expected page state information. The training process can employ a supervised learning paradigm, where the loss function can simultaneously measure the difference between the page operation instructions output by the model and those in the supervision labels, as well as the difference between the expected page state information output by the model and the page state change information in the supervision labels. The fine-tuning process enables the model to learn how to generate formatted and semantically reasonable page operation instructions based on the current page state and task objective, while ensuring high consistency between its predicted expected page state and subsequent actual execution results.

[0106] The decision model obtained through the above fine-tuning process can independently make page operation decisions during inference without relying on external knowledge bases or templates. The page operation knowledge and task execution patterns learned from the training data have been internalized into the model parameters. This allows the fine-tuned decision model to handle diverse page states more directly and adapt to a certain extent to the various page changes covered by the training data.

[0107] In one embodiment of this specification, the decision model built based on the language model can be flexibly deployed on terminal devices or servers according to actual application scenarios.

[0108] Different deployment methods can adapt to varying computing power, network environments, and privacy protection requirements. Terminal deployment reduces network latency and prevents sensitive data from leaving the device; it typically employs compression techniques such as model quantization and distillation to balance inference speed and decision quality. Server deployment leverages ample cloud computing power to run language models with larger parameter scales, achieving stronger inference and generalization capabilities, but requires sending page state information to the server over the network.

[0109] When the decision model is deployed on a server, terminal devices can anonymize page state information before sending it. Anonymization refers to identifying, removing, or replacing potentially sensitive personal data of the user that may be contained in the page state information. For example, semantic analysis of text recognition results and interactive element information can be used to detect whether the page contains sensitive fields such as the user's real name, ID number, bank card number, account balance, transaction details, and home address. Detected sensitive fields can be handled by masking, replacing them with placeholders, or retaining only the field type label without retaining the specific value. The anonymized page state information only retains the page structure and the type and location information of interactive elements, which is sufficient to support the decision model in making page operation decisions without exposing the user's sensitive data to the server.

[0110] The terminal device sends the anonymized page state information and task information to the server via the communication network. Upon receiving this information, the server inputs it into the decision model deployed on it. After completing reasoning, the decision model generates page operation instructions and their corresponding expected page state information, and the server returns these outputs to the terminal device. Upon receiving these outputs, the terminal device's execution control module parses and executes the page operation instructions, and uses the expected page state information for subsequent verification.

[0111] By supporting both terminal and server deployment methods and introducing de-identification processing during server deployment, the risk of leakage of sensitive user information can be reduced while making full use of the cloud-based large model inference capabilities. This allows the system to adapt to devices with different performance requirements and scenarios with different levels of privacy protection.

[0112] In S404, page operation instructions are executed through the operation execution interface to drive the target application to perform page jumps or in-page operations from the current page.

[0113] The operation execution interface is a functional module in the data acquisition client responsible for translating the logical instructions generated by the model into actual operations at the operating system level. This interface receives page operation instructions generated by the decision model and parses them into specific actions for the target application. The actions indicated by these instructions can be actions that trigger page navigation, such as clicking a link or navigation button to switch the target application from the current page to another; or they can be in-page operations, such as entering search keywords in a text box, selecting a checkbox, or scrolling down to view more content. The operation execution interface completes these actions by calling accessibility services provided by the operating system or simulating input from an input device. Through this decoupled design, the decision model only needs to focus on high-level operational intentions without dealing with the complex details of low-level operating system calls.

[0114] In some embodiments, the page operation instructions generated by the decision model can adopt different instruction forms according to the operation type and controllability requirements, so as to balance the execution efficiency of routine operations and the safety and controllability of complex operations.

[0115] Page operation instructions can include tool invocation instructions or template execution instructions. The two types of instructions correspond to different execution paths.

[0116] Tool invocation instructions are structured, high-level semantic commands that do not contain specific low-level operation code. The decision model outputs a description of the target operation tool and its parameters. For example, the model can generate an instruction to invoke a click tool, specifying the target as clicking a button labeled "Query". Upon receiving the tool invocation instruction, the operation execution interface parses it to determine the name of the operation tool to be invoked and its parameters. It then matches the corresponding tool within a pre-defined set of operation tools to complete the actual invocation. The operation tool set can be an encapsulation of the terminal operating system's low-level input interface or an encapsulation of accessibility service interfaces. Through this instruction format, the decision model does not need to generate specific operating system invocation code; it only needs to focus on describing the operation intent, making the model output more concise and stable.

[0117] Template execution instructions are parameterized instructions suitable for complex scenarios where the operation process is relatively fixed but key variables change with the task. Pre-defined code templates are pre-written code frameworks containing the complete execution logic for a specific type of operation, with embedded variable parameter placeholders. For example, in a bank statement application process, date range selection and export operations can be encapsulated as templates, exposing only the start date, end date, etc., as dynamic variables. When the decision model determines that the current operation is suitable for template execution, it generates a template execution instruction containing a template identifier and key variable information. The operation execution interface parses the template execution instruction, extracts the template identifier and key variable information, searches for the corresponding pre-defined code template in the pre-defined template library based on the template identifier, and then fills the key variable information into the corresponding placeholders in the template, generating complete executable code, which is then executed by the execution engine.

[0118] These two instruction formats can be selected as needed. For routine steps in the process, tool call instructions can be used, where the decision model directly specifies the operation tool and target, resulting in a short inference path and high execution efficiency. For complex operations involving precise parameter control, multi-step combinations, or strict adherence to specific business logic, template execution instructions can be used. Pre-set templates ensure the correctness and security of the process, while key variables are populated into the model to adapt to the specific parameter requirements of different task instances.

[0119] In some embodiments, when the automated process reaches a critical node involving information submission, a user confirmation step can be introduced before execution to prevent accidental submissions due to misunderstandings of the page or errors in path planning.

[0120] In data retrieval tasks, some page operation instructions may involve irreversible or high-risk submission actions, such as submitting a data retrieval request, confirming the sending of data to a specific recipient, or submitting critical information. The data retrieval client can enter a confirmation phase before executing the page operation instructions used to submit a data retrieval request or submit critical information.

[0121] The data acquisition client can generate a summary of the content to be submitted based on the form content extracted from the current page status information, text recognition results, and key element information. The summary of the content to be submitted may include the type of data to be submitted, the time range corresponding to the data, recipient information, and key fields that the user needs to confirm.

[0122] After generating the summary of content to be submitted, the data acquisition client can output a confirmation interaction control. This control may include a confirmation pop-up displaying the summary and interactive buttons for the user to confirm or cancel the operation. Once the client detects that the user has confirmed the summary via the confirmation interaction control, it then executes the operation instructions on that page through the operation execution interface. If the user does not confirm or actively cancels within a preset time, the operation instructions are not executed, and the process can be transferred to manual intervention or the decision-making model can re-evaluate subsequent operations.

[0123] By setting up user confirmation at key submission points, the final decision-making power can be retained in the hands of users while preserving the efficiency advantages of automated processes, thereby reducing the risk of erroneous information submission due to biases in automated decision-making.

[0124] In S406, after executing the page operation instruction, the page state information of the target application is retrieved again, and it is determined whether the retrieved page state information matches the expected page state information.

[0125] Each time a page operation is performed, the target application may experience state changes due to network latency, server response, or local processing. To confirm whether the operation has achieved the expected results, the data acquisition client will recapture the screen, perform text recognition and element detection, and regenerate a page state information reflecting the actual environment after the operation.

[0126] For example, this reacquired page state information can be submitted to the decision model and compared with the expected page state information generated in S202. This comparison is not a simple image pixel comparison, but a semantic-level matching performed by the decision model with semantic understanding capabilities. The decision model can determine whether the two page states are consistent in content, structure, and core functional intent. It can not only accurately identify obvious anomalies such as page navigation failures, navigation to incorrect pages, or encountering unexpected pop-ups, but also tolerate harmless dynamic content changes on the page, thereby avoiding misjudging subtle differences in interface rendering as process anomalies and improving the fault tolerance and stability of the entire automation process.

[0127] In other embodiments, the comparison process for determining whether the reacquired page state information matches the expected page state information can also be performed by a verification model independent of the decision model. This verification model also has semantic understanding capabilities and can make semantic matching judgments based on the content, structure, and intent of the page state. Alternatively, the comparison process can also be performed by other trained language models deployed locally or in the cloud. This specification does not limit this process.

[0128] In some embodiments, the timing for re-acquiring page state information after executing a page operation instruction can be dynamically determined based on the operation type and the actual page loading status, rather than using a fixed waiting time. If the fixed waiting time is set too short, the state may be intercepted before the page has finished loading, leading to a misjudgment of a redirect failure; if it is set too long, it will increase unnecessary waiting time and reduce overall execution efficiency.

[0129] If the page operation instruction indicates that the current operation is a page redirection, the expected waiting time for this operation can be determined based on the actual loading time of the target page indicated by the expected page state information in historical execution data. Historical execution data can be statistics on page loading times recorded when performing the same or similar page redirection operations on the same target application in the past. The expected waiting time can be taken as a statistical value of historical loading times, such as the average or a specific quantile.

[0130] After executing page operation commands, you can check whether the target application is in a page loading state through the operation execution interface. The page loading state can be determined by detecting whether a loading progress bar or loading animation appears on the page, or by obtaining the application's network request status through system interfaces.

[0131] If the target application is in a page loading state and the waiting time has not exceeded the expected waiting time, return to the step of checking whether the target application is in a page loading state; if the target application is not in a page loading state, or the waiting time has reached the expected waiting time, re-acquire the page state information of the target application.

[0132] By incorporating a dynamic waiting mechanism based on historical loading data, the waiting time can be adaptively adjusted according to the actual loading characteristics of the target page. This provides sufficient waiting time for slower-loading pages while avoiding unnecessary delays for fast-loading pages. Combined with real-time monitoring of the loading status, the status can be captured immediately after page loading is complete, or the verification process can be initiated promptly in case of a timeout, thus balancing execution efficiency with the accuracy of status acquisition.

[0133] In S408, if the conditions are met, the newly acquired page state information is used as the new current page state information, and the process returns to the steps for executing page operation decisions until the target data acquisition page is reached or the target data acquisition task is completed.

[0134] When the re-acquired page state information is determined to meet expectations, it means that the previous operation has been successfully executed, and the target application's running status has progressed as planned. At this point, the data acquisition client will update this verified latest page state information as the new current page state information. Starting from this new state, the automated process will loop back to the decision step in S202. The decision model will continue to generate the next operation instruction based on this new current state and the persistent task objective. Thus, a closed loop of "perception → decision → execution → verification → re-perception" is formed. This closed-loop approach ensures that even minor deviations at any step will not accumulate and amplify into subsequent steps, enabling the automated process to stably traverse complex multi-step page jumps and ultimately reach the page containing the target data, or directly complete the final operations such as data acquisition, download, or submission.

[0135] In S410, if the condition is not met, the decision model generates a page correction instruction based on the reacquired page state information, expected page state information, and task information, and executes the page correction instruction through the operation execution interface.

[0136] When the reacquired page state information is determined to be inconsistent with expectations, it indicates that the automation process has encountered interference or anomalies. At this point, the process enters the anomaly recovery branch. When generating a correction strategy, the decision model integrates three key pieces of information: the reacquired page state information representing the current real-world environment, the expected page state information representing the correct goal envisioned before the operation, and the top-level task information that runs throughout the process. This multi-information fusion anomaly analysis approach allows the decision model to perceive the specific deviation between the actual state and the expected state, and to determine how to recover from the deviation in conjunction with the overall task objective.

[0137] In one possible implementation, the decision model can autonomously generate correction instructions based on the three types of information mentioned above. The decision model can leverage its understanding of the task objective and its reasoning ability regarding page state semantics to directly deduce appropriate correction actions based on the deviation between the actual state and the desired state, without relying on a predefined set of recovery strategies.

[0138] In another implementation, a preset recovery strategy can be introduced. The preset recovery strategy is a configurable set of operations that provides the decision model with a space for corrective actions. Based on the understanding of the deviation, the decision model selects or combines the most suitable corrective action from this strategy set to generate a page correction instruction. For example, if the deviation is caused by an unexpected pop-up ad, the correction instruction could be to close the pop-up; if the deviation is caused by slow page loading, the correction instruction could be to execute a wait operation. After executing the correction instruction through the operation execution interface, the process can return to the verification step in S206 to verify the correction effect, thus forming a self-recovering closed loop that allows the data acquisition task to continue executing until completion even in a dynamically changing operating environment.

[0139] In some embodiments, when the reacquired page state information differs from the expected page state information, the decision model can first perform a semantic-level difference analysis between the reacquired and expected page state information. This difference analysis may include: determining whether the page intent has changed, identifying newly added or missing interactive elements, and comparing structural changes in the page text content. Based on the results of the difference analysis, the decision model can determine the type of deviation the current page belongs to.

[0140] Page deviation types can include at least one of the following: ① Page does not redirect, i.e., the page content does not change substantially; ② Redirect to an unexpected page, i.e., the page redirects but the target page does not match the expectation; ③ Page loading is abnormal, i.e., the page is in a state of continuous loading or loading failure; ④ Interfering page appears, i.e., the page is covered by unexpected elements such as pop-ups, advertisements, permission request overlays; ⑤ Target interactive elements are missing, i.e. the page intent is correct but expected elements such as buttons and input boxes do not appear.

[0141] After determining the type of page deviation, the decision model can match the corresponding target recovery strategy from the preset recovery strategies based on the deviation type. The preset recovery strategies can be a set of predefined mappings between recovery actions and deviation types. For example, if the page does not redirect, it can be mapped to re-executing the operation command or switching the page entry; if it redirects to an unexpected page, it can be mapped to returning to the previous page or replanning the navigation path; if the page loads abnormally, it can be mapped to waiting for the page to load; if an interfering page appears, it can be mapped to closing or skipping the interfering page, or waiting for the pop-up to close automatically within a time limit; if the target interactive element is missing, it can be mapped to scrolling the page to find it or replanning the navigation path.

[0142] Once a target recovery strategy is matched, the decision model can generate specific page correction instructions based on the recovery strategy, the reacquired page state information, and task information. Page correction instructions include at least one of the following recovery actions: re-execute page operation instructions, return to the previous page, wait for the page to load, close or skip the interfering page, wait for the interfering page with a display time limit to close automatically, scroll the page to find the target interactive element, switch page entry points, and replan the page navigation path.

[0143] For example, the content of a correction instruction may include not only the type of recovery action, but also the specific parameters required to perform that recovery action. For instance, a correction instruction for closing a distracting page may include the element location information of the close button on the distracting page; a correction instruction for scrolling a page to find a target element may include the scrolling direction and a semantic description of the target element.

[0144] By classifying page deviations and pre-setting targeted recovery strategies for each type of deviation, the decision model does not need to reason about recovery solutions from scratch every time an anomaly occurs, which improves the efficiency and success rate of anomaly recovery and enhances the ability of automated processes to cope with various anomalies in complex page environments.

[0145] In some embodiments, if the page state information re-acquired after executing a page correction instruction indicates that the current page has deviated from the page navigation path, the decision model can dynamically update the path. Deviating from the page navigation path means that the current page no longer belongs to the expected page corresponding to any node in the original path; for example, the correction operation might lead the application to an intermediate page not defined in the path. The decision model can re-evaluate the reachable path between the current page and the task objective based on the current page state information, task information, and the page navigation path. The update process may include: starting from the current page, adding or deleting nodes or adjusting the jump order based on the original path, combined with the target constraints in the task information and the information of nodes not yet traversed in the original path; when adjustments cannot be made based on the original path, a new path is replanned. Dynamic path updates allow for timely adjustments to the navigation route after anomaly recovery, improving the process's ability to handle complex anomalies.

[0146] In some embodiments, a manual takeover mechanism can be introduced when the abnormal situation exceeds the automatic recovery capability. Triggering conditions may include: the number of times the page correction instruction is executed reaches a preset number, indicating that the current abnormality cannot be resolved by a predefined recovery strategy; or the decision model determines that the current page state information cannot generate valid page operation instructions, indicating that the current page cannot identify or lacks an executable operation target.

[0147] When one of the above conditions is met, the data acquisition client can pause the automated operation process and output a manual intervention prompt, informing the user of the specific reason for the current task being blocked and the current page status. Simultaneously, a continuation control, such as a floating button, can be rendered on the interface for the user to manually trigger the resumption of the automated process after manual intervention. Once the user completes manual intervention and triggers the continuation control, the data acquisition client re-acquires the page status information and resumes the automated process based on the new status. By setting up a fallback mechanism for exception recovery, the automated process can be prevented from falling into an infinite retry loop, ensuring the user's right to know while maintaining the process's recoverability.

[0148] In some embodiments, automated processes may encounter pages involving user privacy during their execution. These privacy-related pages may include, but are not limited to: requiring users to enter passwords, perform biometric authentication, fill in sensitive personal information, confirm recipient information, or authorize final submission. These operations involve user account security and personal information and should not be performed by automated programs.

[0149] To address this, the ability to identify privacy-sensitive operation pages can be configured in the screen understanding module. When the page state information of any acquired page indicates that the current page is a privacy-sensitive operation page, the data acquisition client can trigger a privacy protection interruption mechanism, pausing the automated operation process and the page state information acquisition function. Pausing the automated operation process means that new operation instructions will not be executed for the time being. Pausing the page state information acquisition function includes stopping the screenshotting of the target application interface and the collection of page state data.

[0150] After a pause, the data acquisition client can generate a prompt message to guide the user to complete the current privacy operation. The prompt message may include instructions on what action the user needs to perform on the current page, such as "Please enter your payment password" or "Please confirm the recipient's information." The prompt message can be displayed as a floating window on the screen, a notification bar message, or a voice announcement. Simultaneously with the prompt message, a continuation control, such as a floating button, can be rendered on the interface for the user to manually trigger the resumption of the automated process after completing the privacy operation.

[0151] Once the continue control is detected as triggered, the data acquisition client can restart the page state information acquisition function, re-capture the screenshot and state of the current target application's interface, and use the re-acquired page state information as the current page state. Based on this new state and task information, the intent decision module continues to execute page operation decisions, driving the automated process to continue from the next page after the privacy operation page.

[0152] Through the aforementioned privacy protection interruption and recovery mechanisms, the continuity of automated processes can be maintained while isolating sensitive user information, enabling data acquisition tasks to proceed securely in scenarios involving multiple steps.

[0153] In some embodiments, the data acquisition client can generate current process progress information based on at least one of the following: current page status information, page operation instructions, expected page status information, and the task progress of the target data acquisition task. For example, the progress ratio can be determined based on the number of steps successfully executed and the total number of steps expected in the task. Alternatively, a semantic progress description can be generated based on the current page intent category, such as having entered the data list page or setting query conditions.

[0154] Once generated, the current process progress information can be output through at least one of the following methods: voice broadcast, text prompts, or a floating pop-up. By outputting this progress information, users can understand the stage of the automated process in real time, distinguish between normal waiting and abnormal stalls during execution, thereby improving the user experience and providing information for users to manually take over when necessary.

[0155] In some embodiments, after completing the target data acquisition task, the data acquisition client can save the complete record of this execution process to the historical trajectory database to provide a reference for path planning and anomaly recovery for subsequent similar tasks.

[0156] After the task is completed, the data acquisition client can record the execution trajectory. The execution trajectory can include at least one of the following: page state information at each step during task execution, page operation instructions, expected page state information, state matching results, and the final task completion result. By recording the complete execution trajectory, the successful execution experience can be transformed into reusable structured knowledge.

[0157] Before recording the execution trajectory, the data acquisition client can first anonymize sensitive content within the trajectory. Anonymization can include identifying sensitive personal data contained in the page status information of each page in the execution trajectory, such as name, ID number, bank card number, account balance, transaction details, and password input text. Identified sensitive fields can be masked, replaced with placeholders, or only field type labels can be retained. The anonymized execution trajectory retains only non-sensitive information such as page structure, the type and location of interactive elements, operation types, and navigation logic, sufficient to support subsequent path reuse and decision-making reference.

[0158] The anonymized execution trajectory is stored in a historical trajectory database. The historical trajectory database can be categorized and stored according to dimensions such as data acquisition task type, target application, and execution scenario information, so that when executing similar tasks in the future, referenceable execution trajectories can be quickly retrieved through scenario matching, assisting the decision-making model in path planning or anomaly recovery.

[0159] In one exemplary application scenario, the scenario involves a user applying for personal social security participation certificate through a government service platform, covering five stages: task triggering, authorization, navigation, information filling, and completion confirmation.

[0160] Phase 1: Task Initialization and Authorization Confirmation Step 1.1: Receive user applications On the business processing page of the government service application, users select the item "Apply for Personal Social Security Participation Certificate". The system displays a list of supported social security data provision channels, such as the "XX Municipal Human Resources and Social Security Bureau" and the "XX Government Service Website" and other corresponding applications.

[0161] Step 1.2: Obtain explicit authorization from the user After the user selects the target application, the system pops up an authorization interface, clearly informing the user that: the [social security service application name] will be automatically opened and your social security participation certificate will be obtained under your authorization; your login password and other sensitive information will not be saved or transmitted; the operation can be paused or stopped at any time.

[0162] After the user reads the prompt and clicks "Agree and Continue," the system obtains authorization to proceed to the next stage.

[0163] Phase 2: Environment Preparation and Application Launch Step 2.1: Inspect the target application The data acquisition client checks if the target application is installed on the terminal device. If not, it outputs a prompt message guiding the user to download and install it. If already installed, it further checks the version status of the target application, outputting a version update prompt if the version is too old and may be incompatible. After confirming that it is installed and the version meets the execution conditions, it proceeds to the next step.

[0164] Step 2.2: Initialize the data acquisition client The data acquisition client requests accessibility service permissions, obtains the task information for the target data acquisition task (see the description above for specific task information), and initializes the text recognition model and interface element detection model in the screen understanding module.

[0165] Step 2.3: Launch the target application The data acquisition client launches the target application through the operating system interface, captures a screenshot of the homepage after launch, and identifies the current page status, such as whether it is in an unlogged-in state or the homepage of a logged-in user.

[0166] Phase 3: Automated navigation to the target page Step 3.1: Page Status Recognition Each time the page changes, the data acquisition client performs the following perception operations: captures a screenshot of the current screen; identifies the text content on the page using a text recognition model; detects interactive elements on the page using an interface element detection model; and classifies the intent of the current page using a page intent recognition model. Based on the above perception results, it determines whether the target page, namely the social security participation certificate application page, has been reached.

[0167] Step 3.2: Decide on the next step. The intent decision module makes page operation decisions based on the current page status and task information. If the current page is a login page, the automated process is paused and the user is prompted to complete the login process manually. If the current page is the application homepage, the decision model generates an operation instruction to find and click the "Personal Center" or "Social Security Services" entry. If the current page is the Personal Center page, it generates an operation instruction to find and click the "Participation Certificate" or "Social Security Certificate Printing" entry. If the current page is a certificate application page, it generates an operation instruction to locate and click the "Apply" or "Download" button.

[0168] Step 3.3: Perform the operation The execution control module performs corresponding click or swipe operations based on page operation instructions via the accessibility service interface. After the operation is executed, it waits for a page response. By re-capturing the page and matching it with the expected page state information, it verifies whether the operation successfully brought the application to the expected page.

[0169] Step 3.4: Exception Handling When an unexpected pop-up is detected, the decision model generates instructions to close the pop-up or skip the interruption based on the pop-up's page intent. When page loading is continuously incomplete, a preset waiting strategy is applied. If the target element cannot be located multiple times consecutively, the automated process is paused, and the user is shown the current page position, allowing the user to decide whether to intervene manually.

[0170] Phase 4: Key Information Confirmation Phase When the system navigates to the social security participation certificate application page and requires the user to confirm the receiving method or application information, the data acquisition client pauses the automated operation process. A prompt message is displayed on the interface through the user interaction module, such as: "Your social security participation certificate is about to be generated. Please confirm the application information." A confirmation interaction control is also output.

[0171] After detecting that the user has confirmed the request through the confirmation interaction control, the data acquisition client executes the page operation instructions for submitting the application through the operation execution interface.

[0172] Phase 5: Task Completion and Cleanup Step 5.1: Confirm successful operation The data acquisition client identifies the page indicating successful application submission or complete download of supporting documents, captures the success page as proof of operation completion, and outputs a task completion prompt through the user interaction module.

[0173] Step 5.2: Resource Release The data acquisition client closes the requested accessibility service permissions, clears temporary cached data generated during task execution, and returns the business process of the government affairs application.

[0174] The various technical features in the above embodiments can be combined arbitrarily, as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, the arbitrary combination of various technical features in the above embodiments is also within the scope of this specification.

[0175] Figure 5 This is a schematic structural diagram of a device provided in an exemplary embodiment. For example... Figure 5 As shown, device 500 mainly consists of a communication interface 502, a user interface 504, a processor 506, and a data storage 508. These components are interconnected and communicate with each other via a system bus, network, or other connection mechanism 510. The communication interface 502 enables device 500 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface 502 may include an antenna and related processing devices for wireless communication with a radio access network or access point. Furthermore, the communication interface 502 can be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, the communication interface 502 can also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface 502 may also include multiple physical communication interfaces, such as Wi-Fi interfaces, Bluetooth interfaces, and wide-area wireless interfaces.

[0176] User interface 504 includes receiving user input and providing output to the user. Therefore, user interface 504 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. User interface 504 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, user interface 504 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, device 500 may support remote access from other devices via communication interface 502 or another physical interface (not shown). User interface 504 may be configured to receive user input, the position and movement of which may be indicated by indicators or cursors described herein. User interface 504 may also be configured as a display device for rendering or displaying text fragments.

[0177] Processor 506 may contain one or more general-purpose processors and / or special-purpose processors.

[0178] Data storage 508 may include one or more volatile and / or non-volatile storage components and may be integrated wholly or partially with processor 506. Data storage 508 may include removable and non-removable components.

[0179] Processor 506 is capable of executing program instructions 518 (e.g., compiled or uncompiled program logic and / or machine code) stored in data storage 508 to perform the various functions described herein. Data storage 508 may contain a non-transitory computer-readable medium on which program instructions are stored, which, when executed by device 500, enable device 500 to perform any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Execution of program instructions 518 by processor 506 may result in processor 506 using data 512.

[0180] For example, program instructions 518 may include an operating system 522 (e.g., an operating system kernel, device drivers, and / or other modules) installed on device 500 and one or more applications 520 (e.g., a browser, social application, or game application). Similarly, data 512 may include operating system data 516 and application data 514. Operating system data 516 is primarily accessible to the operating system 522, while application data 514 is primarily accessible to one or more applications 520. Application data 514 may reside in a file system visible or hidden from the user of device 500.

[0181] Application 520 can communicate with operating system 522 through one or more application programming interfaces (APIs). These APIs help application 520 read and / or write application data 514, transmit or receive information via communication interface 502, receive or display information on user interface 504, etc.

[0182] In some terminology, application 520 may be simply referred to as "app". Furthermore, application 520 can be downloaded to device 500 through one or more online app stores or app markets. However, applications can also be installed on device 500 in other ways, such as through a web browser or a physical interface on device 500 (e.g., a USB port).

[0183] In some embodiments, the target data automatic acquisition device can be applied to, for example... Figure 5 The specific details of the automatic target data acquisition device used to implement the technical solution described in this specification can be found in the description of the above method, and will not be repeated here.

[0184] Based on the same concept as the methods described above, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in any of the above embodiments by executing the executable instructions.

[0185] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0186] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0187] What those skilled in the art will understand is: In this specification, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded.

[0188] In this specification, “a,” “an,” and “the” do not specifically refer to the singular, but may also include the plural.

[0189] In this specification, ordinal numbers such as "first," "second," etc., do not necessarily indicate order; they are often used to distinguish between objects. For example, "first server" and "second server" usually refer to two servers. To differentiate between these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.

[0190] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0191] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0192] This specification uses specific terms to describe embodiments thereof. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples, without contradiction.

[0193] Although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is only one of many possible execution orders and does not represent the only execution order. Therefore, when the claims involve method steps, any changes or adjustments to the order of such steps, or the parallelism between steps, are also within the scope of protection of the claims.

Claims

1. A method for automatically acquiring target data, comprising: Obtain target data, obtain task information for the task, and obtain the current page status information of the target application to which the task information is applied; By using a decision model built on a language model, page operation decisions are made based on the current page state information and the task information, generating the next page operation instruction to be performed and its corresponding expected page state information. The page operation instructions are executed through the operation execution interface to drive the target application to perform page jumps or in-page operations from the current page; After executing the page operation instruction, the page state information of the target application is reacquired, and it is determined whether the reacquired page state information matches the expected page state information. If the conditions are met, the newly acquired page state information will be used as the new current page state information, and the process will return to the steps of executing the page operation decision until the target data acquisition page is reached or the target data acquisition task is completed. If the condition is not met, the decision model generates a page correction instruction based on the reacquired page state information, the expected page state information, and the task information, and executes the page correction instruction through the operation execution interface.

2. The method according to claim 1, wherein the step of acquiring target data and acquiring task information includes: Obtain basic task information from the application that initiated the task of obtaining the target data; Obtain a set of page description information containing at least one page for the target application in relation to the target data acquisition task from a preset application description library; wherein, the application description library is used to store page description information of each relevant page for different applications in relation to different types of data acquisition tasks; The decision model built based on a language model makes page operation decisions based on the current page state information and the task information, generating page operation instructions and their corresponding expected page state information, including: In the absence of a page navigation path, the decision model plans a page navigation path from the current page to the target data acquisition page based on the basic task information, the page description information set, and the current page state information. Based on the page navigation path and the current page state information, path reasoning is performed to generate the page operation instructions and their corresponding expected page state information. Given that the page navigation path has been formed, the decision model performs path reasoning based on the current page state information and the page navigation path to generate the page operation instructions and their corresponding expected page state information.

3. The method according to claim 2, wherein the step of acquiring target data and task information further includes: Obtain the general page navigation path corresponding to the target data acquisition task from the preset process template library; wherein, the process template library is used to store general page navigation paths for different types of data acquisition tasks; The step of planning a page navigation path from the current page to the target data acquisition page using the decision model, based on the basic task information, the page description information set, and the current page state information, includes: Using the decision model, the general page navigation path corresponding to the target data acquisition task is adjusted based on the basic task information, the page description information set, and the current page state information to obtain the page navigation path.

4. The method according to claim 2 or 3, wherein the step of acquiring target data and task information further includes: From a preset historical trajectory database, retrieve the historical execution trajectory and its execution scenario information corresponding to the target data acquisition task; wherein, the historical trajectory database is used to store the historical execution trajectory and its corresponding execution scenario information for different types of data acquisition tasks; The method further includes: If the page navigation path has not yet been formed, or the page navigation path that has been formed cannot advance to the target data acquisition page, if the historical execution trajectory corresponding to the target data acquisition task is successfully obtained, and the execution scenario information of the historical execution trajectory matches the current execution scenario indicated by the current page status information, the decision model determines the page navigation path from the current page to the target data acquisition page from the successfully obtained historical execution trajectory based on the current page status information.

5. The method according to claim 1, wherein the decision model constructed based on the language model is a model obtained by fine-tuning the language model based on several training samples; The training samples include page state information and task information as input to the model, as well as page operation instructions and their corresponding expected page state information as supervision labels. in, The training samples are obtained by processing the operation trajectories of different applications for different data acquisition tasks collected in advance. The operation trajectory includes the page state information of each page involved in the corresponding data acquisition task, the page operation behavior executed in the page state corresponding to the page state information, and the page state change information formed after the page operation behavior is executed.

6. The method according to claim 1, wherein the page state information of any page includes at least one of the following: The text recognition result obtained by performing text recognition on the screen image of the page; Interactive element information obtained by detecting interface elements in the screen image, wherein the interactive element information includes at least one of the following: element type, display position, interactive attributes, and element text; Page intent information is obtained by performing page intent recognition based on the screen image, the text recognition result, and / or the interactive element information. Key element information selected from the interactive element information that is relevant to the target data acquisition task.

7. The method according to any one of claims 1 to 3, wherein generating a page correction instruction through the decision model based on the reacquired page state information, the expected page state information, and the task information comprises: The decision model determines the page deviation type based on the difference between the reacquired page state information and the expected page state information. Based on the page deviation type, a corresponding target recovery strategy is determined from the preset recovery strategies. The page correction instruction is then generated based on the target recovery strategy, the reacquired page state information, and the task information.

8. The method according to claim 7, wherein the page deviation type includes at least one of the following: page not redirecting, redirecting to an unexpected page, page loading error, appearance of an interfering page, and missing target interactive element; The page correction instructions include at least one of the following: re-execute the page operation instructions, return to the previous page, wait for the page to load, close or skip the interfering page, wait for the interfering page with a display time limit to close automatically, scroll the page to find the target interactive element, switch the page entry, and replan the page navigation path.

9. The method according to claim 1, further comprising: If the page status information of any page indicates that the page requires the user to perform a privacy operation, pause the automated operation process and pause the page status information acquisition function, output a prompt message to prompt the user to complete the privacy operation and a continue control to trigger the resumption of the automated operation process. If the continue control is detected to be triggered, the page state information acquisition function is restarted to reacquire the page state information of the target application, and the automated operation process is continued based on the reacquired page state information.

10. The method according to claim 1, wherein the page operation instructions include tool invocation instructions or template execution instructions; The execution of the page operation instructions through the operation execution interface includes: If the page operation instructions include tool call instructions, the operation tool corresponding to the tool call instructions is called through the operation execution interface to execute the tool call instructions; If the page operation instruction includes a template execution instruction, the template execution instruction is parsed to obtain a template identifier and key variable information. A preset code template is determined based on the template identifier, and the key variable information is filled into the preset code template to generate executable code. The executable code is then executed through the operation execution interface.

11. The method according to claim 1, wherein after executing the page operation instruction, re-acquiring the page state information of the target application includes: If the page operation instruction indicates that the current page operation is a page jump operation, the expected waiting time corresponding to the current page operation is determined based on the actual loading time of the page in the historical execution data indicated by the expected page status information; After executing the page operation instruction, it is detected whether the target application is in a page loading state; If the target application is in a page loading state and the waiting time has not exceeded the expected waiting time, return to the step of detecting whether the target application is in a page loading state; If the target application is not in a page loading state, or if the waiting time has reached the expected waiting time, the page status information of the target application is retrieved again.

12. An electronic device, comprising: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method as described in any one of claims 1-11 by executing the executable instructions.

13. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1-11.

14. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1-11.

Images