Mobile terminal task automatic execution method, device, equipment and storage medium

Abstract

The application discloses a mobile terminal task automatic execution method and device, equipment and storage medium, relates to the mobile terminal automation technical field, and includes: in response to the task automatic execution instruction of the mobile terminal, the operation intention graph pre-constructed of the target task is acquired, wherein the operation intention graph is obtained based on the modeling of the multi-modal operation data recorded by the mobile terminal; in the current display interface of the mobile terminal, real-time interface element structure information is acquired; the target operation element of the target task is determined in the interface element structure information based on the operation intention graph; the target task is automatically executed based on the target operation element, the task execution is guided through the pre-constructed operation intention graph, the high delay and high cost caused by real-time reasoning are avoided, the real-time interface element structure information acquisition and dynamic matching mechanism improve the self-adaptation capability of the task in cross-version and cross-device execution. The combination of the two improves the efficiency and robustness of the mobile terminal task automation.

Description

Technical Field

[0001] This application relates to the field of mobile automation technology, and in particular to methods, apparatus, devices and storage media for automatic execution of mobile tasks. Background Technology

[0002] With the widespread adoption of mobile office solutions in enterprises, a large number of repetitive and procedural tasks still rely on manual completion, which seriously consumes human resources and restricts business efficiency.

[0003] Currently, the field of mobile task automation mainly includes several technical solutions such as script programming, real-time screenshot decision-making, and PC-based remote control. However, script programming-based solutions require users to have professional programming skills and have high maintenance costs; multimodal intelligent agent solutions based on real-time screenshot decision-making require calling a large model frame by frame for real-time inference each time a task is executed, resulting in high inference latency and high costs; PC-based RPA remote control solutions cannot run independently on the mobile phone, are greatly affected by network latency, and have poor stability. Summary of the Invention

[0004] The main purpose of this application is to provide a method, apparatus, device and storage medium for automatic execution of mobile tasks, which aims to solve the technical problems of poor execution effect and high cost of current mobile task automation solutions.

[0005] To achieve the above objectives, this application proposes a method for automatically executing mobile tasks, the method comprising: In response to the task auto-execution command from the mobile terminal, a pre-constructed operation intent map of the target task is obtained, wherein the operation intent map is modeled based on multimodal operation data recorded by the mobile terminal; Obtain real-time information about the structure of UI elements on the current display screen of the mobile device; Based on the operation intent map, the target operation element of the target task is determined in the interface element structure information; The target task is executed automatically based on the target operation element.

[0006] In one embodiment, the step of determining the target operation element of the target task based on the operation intent map in the interface element structure information includes: Based on the operation intent map, a structured operation semantic annotation sequence for the target task is obtained; Based on the structured operation semantic annotation sequence, a multi-level degradation strategy is used to determine the target operation element of the target task from the interface element structure information.

[0007] In one embodiment, the multi-level degradation strategy includes: a first-level matching; The step of determining the target operation element of the target task from the interface element structure information using a multi-level degradation strategy based on the structured operation semantic annotation sequence includes: Extract the resource identifier, text content, and control class name of the pre-stored target element from the structured operation semantic annotation sequence; The resource identifier, the text content, and the control class name are used to perform a first-level match in the interface element structure information; When the first-level matching is successful, the node in the interface element structure information that is the same as the resource identifier, the text content, and the control class name is taken as the target operation element of the target task.

[0008] In one embodiment, the multi-level degradation strategy further includes: a second-level fuzzy matching; After the step of performing a first-level match on the interface element structure information using the resource identifier, the text content, and the control class name, the method further includes: If the first-level matching fails, obtain the relative position constraints of the target task; The text content is subjected to a second-level fuzzy matching, and reference elements are determined based on the relative position constraints. Candidate nodes located within the preset orientation area of ​​the reference elements are filtered from the interface element structure information. Candidate nodes that meet the fuzzy matching conditions and relative position constraints are identified as the target operation elements of the target task.

[0009] In one embodiment, the multi-level degradation strategy further includes: a third-level matching; After the steps of performing a second-level fuzzy matching on the text content, determining reference elements based on the relative position constraints, and filtering candidate nodes located within the preset orientation area of ​​the reference elements in the interface element structure information, the method further includes: If the second-level matching fails, capture the screen image of the currently displayed interface on the mobile device; The screen image and the semantic description in the structured operation semantic annotation sequence are input into the visual reasoning model for third-level matching to obtain the target operation elements of the target task.

[0010] In one embodiment, the step of pre-constructing the operation intent graph includes: Acquire multimodal operation data recorded on the mobile device; Cross-modal semantic alignment is performed based on the multimodal operation data to obtain a structured operation semantic annotation sequence; An operation intent graph is obtained by modeling the structured operation semantic annotation sequence.

[0011] In one embodiment, the step of performing cross-modal semantic alignment based on the multimodal operation data to obtain a structured operation semantic annotation sequence includes: Calculate the inter-frame difference index between adjacent frames in the multimodal operation data; Detect screen transition points based on the inter-frame difference index; Based on the screen transition point, the video between adjacent transition frames is used as the operation atom; Perform cross-modal alignment of vision and control for each of the aforementioned operation atoms, and construct a multi-level mapping table; A structured operation semantic annotation sequence is generated based on the multimodal operation data and the multilevel mapping table.

[0012] In one embodiment, the step of generating a structured operation semantic annotation sequence based on the multimodal operation data and the multi-level mapping table includes: Extract the front screen state snapshot, touch action parameters, control operation snapshot, and back screen state snapshot of each operation atom from the multimodal operation data; Obtain the speech intent annotation from the multimodal operation data; The front screen state snapshot, touch action parameters, control operation snapshot, and rear screen state snapshot are encapsulated into a quadruple of operation atoms; The quadruple data, the speech intent annotation, and the multi-level mapping table are input into a multimodal large language model for intent reasoning, generating a structured operation semantic annotation that includes operation intent, precondition assertion, and poststate assertion.

[0013] In one embodiment, the step of performing cross-modal alignment of vision and control on each of the operation atoms to construct a multi-level mapping table includes: Obtain the touch hotspot area in the video frame corresponding to the operation atom; The control tree and touch coordinates are obtained based on the multimodal operation data; Traverse the leaf nodes in the control tree and filter the nodes whose control properties and touch coordinates intersect in space as candidate control nodes; Text recognition is performed on the touch hotspot area to obtain the text recognition result; When the text attribute of the candidate control node matches the result of the text recognition, the corresponding candidate control node is taken as the target control node, and a three-level mapping table containing the mapping relationship between visual pixel regions, target control node identifiers, and semantic tags is generated.

[0014] In one embodiment, the step of acquiring multimodal operation data recorded by the mobile device includes: Acquire screen video streams, touch trajectory streams, control snapshot streams, environmental status data of the mobile device, and voice input data recorded by the mobile device; Convert the voice input data into voice intent annotations; The screen video stream, the touch trajectory stream, the control snapshot stream, the environmental state data, and the voice intent annotation are frame-level aligned and encapsulated using a unified timestamp protocol to generate enhanced video encapsulation data. Multimodal operation data is obtained based on the enhanced video encapsulation data.

[0015] In one embodiment, the step of automatically executing the target task based on the target operation element includes: Based on the coordinate range of the target operation element, generate anthropomorphic touch events containing Bézier curve trajectories and random speed fluctuations; The anthropomorphic touch events are injected into the touch driver layer of the mobile device to automatically execute the target task.

[0016] In one embodiment, after the step of automatically executing the target task based on the target operation element, the method further includes: Obtain screenshot data of operations during the execution of the target task; Perform a closed-loop verification between the screenshot data and the post-state assertions in the operation intent map; When the closed-loop verification result exceeds the preset tolerance, the target task execution is determined to be abnormal, and the automatic healing mechanism is triggered. The target task is restored based on the aforementioned automatic healing mechanism.

[0017] Furthermore, to achieve the above objectives, this application also proposes a mobile task auto-execution device, which includes: The acquisition module is used to acquire a pre-built operation intent map of the target task in response to the task auto-execution command of the mobile terminal, wherein the operation intent map is obtained by modeling based on multimodal operation data recorded by the mobile terminal; The acquisition module is also used to acquire real-time interface element structure information on the current display interface of the mobile terminal. The determination module is used to determine the target operation element of the target task in the interface element structure information based on the operation intent map; An execution module is used to automatically execute the target task based on the target operation element.

[0018] In one embodiment, the determining module is further configured to obtain a structured operation semantic annotation sequence for the target task based on the operation intent map; Based on the structured operation semantic annotation sequence, a multi-level degradation strategy is used to determine the target operation element of the target task from the interface element structure information.

[0019] In one embodiment, the multi-level degradation strategy includes: a first-level matching; the determining module is further configured to extract the resource identifier, text content, and control class name of the target element pre-stored in the structured operation semantic annotation sequence; The resource identifier, the text content, and the control class name are used to perform a first-level match in the interface element structure information; When the first-level matching is successful, the node in the interface element structure information that is the same as the resource identifier, the text content, and the control class name is taken as the target operation element of the target task.

[0020] In one embodiment, the multi-level degradation strategy further includes: a second-level fuzzy matching; the determining module is further configured to obtain the relative position constraints of the target task when the first-level matching fails. The text content is subjected to a second-level fuzzy matching, and reference elements are determined based on the relative position constraints. Candidate nodes located within the preset orientation area of ​​the reference elements are filtered from the interface element structure information. Candidate nodes that meet the fuzzy matching conditions and relative position constraints are identified as the target operation elements of the target task.

[0021] In one embodiment, the multi-level degradation strategy further includes: a third-level matching; the determining module is also used to capture a screen image of the currently displayed interface of the mobile terminal when the second-level matching fails. The screen image and the semantic description in the structured operation semantic annotation sequence are input into the visual reasoning model for third-level matching to obtain the target operation elements of the target task.

[0022] In one embodiment, the device further includes a construction module, which is used to acquire multimodal operation data recorded by a mobile device; Cross-modal semantic alignment is performed based on the multimodal operation data to obtain a structured operation semantic annotation sequence; An operation intent graph is obtained by modeling the structured operation semantic annotation sequence.

[0023] In addition, to achieve the above objectives, this application also proposes a mobile task auto-execution device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the mobile task auto-execution method as described above.

[0024] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the mobile terminal task automatic execution method described above.

[0025] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the mobile task automatic execution method described above.

[0026] This application proposes one or more technical solutions that, in response to a task auto-execution command from a mobile terminal, acquire a pre-built operation intent map of the target task, wherein the operation intent map is modeled based on multimodal operation data recorded on the mobile terminal; acquire real-time interface element structure information on the current display interface of the mobile terminal; determine the target operation element of the target task based on the operation intent map in the interface element structure information; and automatically execute the target task based on the target operation element. By guiding task execution with the pre-built operation intent map, the high latency and high cost of real-time inference are avoided. The real-time acquisition of interface element structure information and the dynamic matching mechanism improve the task's adaptability across versions and devices. The combination of these two aspects enhances the efficiency and robustness of mobile task automation. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0028] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a flowchart illustrating an embodiment of the mobile task auto-execution method of this application. Figure 2 This is a flowchart illustrating Embodiment 2 of the mobile task auto-execution method of this application; Figure 3 This is a flowchart illustrating Embodiment 3 of the mobile task auto-execution method of this application; Figure 4 A simplified flowchart is provided for one embodiment of the mobile terminal task automatic execution method of this application; Figure 5This is a schematic diagram of the module structure of the mobile task automatic execution device according to an embodiment of this application; Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the mobile task automatic execution method in the embodiments of this application.

[0030] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0031] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0032] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0033] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or mobile task auto-execution device capable of the above functions. The following description uses a mobile task auto-execution device as an example to illustrate this embodiment and the subsequent embodiments. All actions involving the acquisition of signals, information, or data in this application are performed in accordance with the relevant data protection regulations of the country where the application is located and with authorization from the owner of the corresponding device.

[0034] Based on this, embodiments of this application provide a method for automatic execution of mobile tasks, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the mobile task auto-execution method of this application.

[0035] In this embodiment, the mobile task automatic execution method includes steps S10 to S40: Step S10: In response to the task auto-execution command from the mobile terminal, obtain the pre-built operation intent map of the target task, wherein the operation intent map is modeled based on multimodal operation data recorded by the mobile terminal.

[0036] It should be noted that the execution entity in this embodiment is a mobile task auto-execution device. This device architecture consists of three core modules: an enhanced recording module, an AI learning engine, and an autonomous execution engine. The enhanced recording module is a recording engine integrated into the Android or iOS system layer of the mobile device. It can capture the screen of the mobile device and simultaneously collect metadata from multiple channels to obtain multimodal operation data. The AI ​​learning engine is a private multimodal AI service cluster deployed on the enterprise intranet, used for in-depth analysis of multimodal operation data to understand the user's operation intent and model an operation intent graph. The autonomous execution engine is an execution engine running on the mobile device. It can obtain the current screen control tree based on the Accessibility Service, locate the target element through a semantic fingerprint three-level degradation strategy, execute the operation in conjunction with a human-like touch behavior engine, and verify the execution result through visual closed-loop verification after each operation. Accessibility services allow authorized applications to listen to system-level events (interface changes, focus switching, etc.) and obtain the hierarchical structure tree of all UI controls on the current screen. They also support performing simulated operations on controls (clicking, scrolling, text input, etc.). This embodiment uses this service as the execution engine for task automation and the screen state awareness channel.

[0037] An Operation Intent Graph (OIG) is essentially a structured task model data that describes user tasks in the form of a Directed Acyclic Graph (DAG). In this DAG, nodes represent screen states, edges represent actions, and it supports conditional branches, loop structures, and exception handling paths. The OIG is the core carrier for task learning, storage, sharing, and execution. Unlike existing solutions that rely on real-time screenshots for decision-making, the OIG in this embodiment is pre-built before task execution. During the construction phase, multimodal data, including screen video, touch trajectories, control structures, and environmental data, is collected by recording user demonstrations. This data is then processed through cross-modal semantic alignment and intent reasoning to transform the user's actions into a machine-understandable structured semantic sequence. This OIG not only records the action sequence (e.g., "click the submit button") but also includes preconditional assertions (e.g., "the submit button is visible") and postconditional assertions (e.g., "the page redirects to the success page"). This pre-built approach eliminates the need for real-time reasoning from large cloud models during execution, significantly reducing execution latency and computational costs. The structured model also facilitates auditing and traceability.

[0038] The automatic task execution command on the mobile device can be triggered by the user clicking the automated task button, or it can be automatically invoked by preset timed tasks and trigger conditions. The automatic task execution command carries the identification information of the target task to be executed, and the corresponding pre-built operation intent map can be retrieved based on the identification information.

[0039] Step S20: Obtain real-time interface element structure information on the current display interface of the mobile device.

[0040] In practical implementation, interface element structure information refers to the hierarchical description data of all UI controls on the current screen, such as the Accessibility NodeInfo tree in the Android system or the UI control element tree in the iOS system.

[0041] Specifically, the interface element structural information includes attributes such as resource identifiers, text content, class names, coordinate boundaries, and hierarchical relationships for each control. This structural information is captured in real-time by the autonomous execution engine calling system-level interfaces (such as accessibility services), serving as the "perceptual basis" for subsequent element positioning. This perception method, based on structural information rather than pixel coordinates, enables the device to adapt to changes in interface layout. For example, when an application version update causes a button's position to shift, the device can still accurately identify it as long as its structural attributes remain essentially unchanged.

[0042] Step S30: Based on the operation intent map, determine the target operation element of the target task in the interface element structure information.

[0043] In practical implementation, the specified target element can be quickly located in the interface element result information based on the data in the operation intent map. The operation intent map stores the semantic description of the operation required for the target task (such as "click the button with ID btn_submit"), while the interface element structure information provides the real-time control tree of the current screen. A semantic matching strategy can be used to find nodes in the real-time control tree that match the semantic description in the intent map. For example, if the intent map indicates that a "agree" button needs to be clicked, then a button control with the text attribute "agree" is searched in the current interface structure. This embodiment is not limited to a single exact match, but supports a multi-level degradation strategy to ensure that the target element can still be accurately located even when the interface undergoes minor changes (such as increased text spaces or dynamic ID changes), thereby ensuring the robustness of execution.

[0044] Step S40: Automatically execute the target task based on the target operation element.

[0045] Understandably, after identifying the target element, corresponding execution instructions can be generated based on the operation type (such as click, swipe, input) defined in the operation intent map. These instructions are not simply coordinate replays, but rather generate touch events based on the real-time coordinate range of the target element and execute them by injecting them into the driver layer. For example, if the target element is a "confirm" button, the system will calculate the button's real-time coordinate area on the current screen and generate a simulated click event within that area. Furthermore, to circumvent application anti-automation detection mechanisms, the execution process can incorporate human-like processing, controlling the human-like touch behavior engine within the autonomous execution engine to begin operation, making the machine's operation statistically closer to human operation.

[0046] In one feasible implementation, step S40 may include steps A11-A12: Step A11: Based on the coordinate range of the target operation element, generate an anthropomorphic touch event containing a Bézier curve trajectory and random speed fluctuations; It should be noted that due to the inherent randomness and non-linearity of finger touch operations, the trajectory of finger movement is not a strictly straight line, but a curve with curvature; the touch speed is not constant, but changes non-linearly. Therefore, this embodiment uses a parametric random model to simulate these characteristics. First, the coordinate range of the target operation element is obtained, and a touch starting point is randomly selected within this boundary, rather than fixing the center of the clicked element, to simulate the slight deviation of human click position. Second, a touch trajectory is generated, which is not a simple straight line, but is generated using a third-order Bézier curve. By introducing randomly generated control points, each generated sliding trajectory has a unique curvature and degree of curvature, thereby simulating the natural jitter and curvilinear characteristics of human finger movement. At the same time, the touch speed exhibits a bell-shaped distribution, superimposed with random speed fluctuations. Specifically, the time axis of the touch event is not non-linearly distributed, but generated based on a bell-shaped distribution (such as a normal distribution) model, making the touch action faster in the middle stage and slower at the beginning and end stages, and superimposing a random time offset on the base duration to avoid all operations having the same time interval. By combining the aforementioned Bezier curve trajectory with random speed fluctuations, the generated touch events possess random and nonlinear characteristics in multiple dimensions such as trajectory shape, speed distribution, and time interval, thus statistically approximating real human operations. Furthermore, by incorporating normally distributed random waiting times into adjacent operations, not only is operational reliability ensured, but anti-automation detection applications can also be effectively avoided.

[0047] Step A12: Inject the anthropomorphic touch event into the touch driver layer of the mobile terminal to automatically execute the target task.

[0048] After generating anthropomorphic touch events, these events are injected into the touch driver layer via the system's low-level interface. This low-level injection method simulates the input signals generated by the touchscreen hardware. From the application layer's perspective, this operation is completely consistent with the signals generated by a user's finger actually touching the screen, thus effectively circumventing the application layer's detection of automated tool interface calls. For example, in the Android system, touch event streams are injected through the `dispatchGesture` interface of `AccessibilityService` or the lower-level `InputManager`. This event stream contains the previously generated sequence of Bézier curve trajectory coordinates and the corresponding timestamp sequence. The driver layer controls the screen response according to this sequence to complete operations such as clicking and swiping. By combining low-level injection with anthropomorphic features, not only is the accurate execution of operations guaranteed, but the stealth and success rate of automated tasks in application environments with anti-fraud capabilities are also significantly improved.

[0049] In practice, simply completing a click or swipe action does not guarantee the task's success. Operations may fail due to network latency, unexpected pop-ups, application crashes, or other abnormal situations. Therefore, after step S40, steps S41-S44 are also included: Step S41: Obtain screenshot data of the operation during the execution of the target task; After injecting touch events through a human-like touch engine, the engine automatically captures a screenshot of the screen after each operation. This screenshot data serves as an objective basis for verification, reflecting the actual interface state after the operation. For example, after performing the "click submit" operation, the captured screen image may display a "submission successful" message, a "network error" pop-up, or remain unchanged on the original page.

[0050] Step S42: Perform a closed-loop verification between the operation screenshot data and the post-state assertion in the operation intent map; In practice, the operation intent map not only records the operation actions but also pre-defines the post-state assertions corresponding to each operation atom. These post-state assertions are generated through multimodal large language model reasoning during the intent map construction phase and describe the expected state that the interface should present after a successful operation, such as "page redirects to the details page," "a 'save successful' Toast message appears," or "the input box content is cleared." The closed-loop verification process involves comparing the operation screenshot data with this expected state.

[0051] The comparison dimensions can include the presence of key text, the similarity of the control tree structure, and pixel changes in specific areas. For example, if the post-state assertion is "the text 'submission successful' appears," then OCR recognition is performed on the screenshot to determine if the text exists; if the assertion is "page redirection," then the structural similarity index (SSIM) between the current screenshot and the expected page is calculated. If the comparison results are highly consistent, the verification is considered successful; if the difference exceeds the preset tolerance, the verification is considered unsuccessful. This semantic assertion-based verification method, unlike the simple coordinate comparison in existing technologies, can accurately determine whether the operation has truly taken effect from a functional perspective, effectively identifying hidden anomalies such as "false success."

[0052] Step S43: When the closed-loop verification result is greater than the preset tolerance, determine that the target task execution is abnormal and trigger the automatic healing mechanism; Understandably, the preset tolerance can be set in advance; it's a threshold used to measure the degree of deviation between the actual state and the expected state. When the closed-loop verification result exceeds this tolerance, it means the operation has not achieved the expected effect. At this point, the operation screenshot data and operation intent map can be input into a multimodal large language model to determine whether the page functions are equivalent. If they are not equivalent, the current step is determined to be abnormal. In this case, instead of directly reporting an error and stopping, an automatic healing mechanism is triggered. The automatic healing mechanism includes trying alternative operation paths, controlling the mobile device to scroll the screen to find the target element, or controlling the mobile device to return to the previous page to retry.

[0053] Step S44: Perform target task anomaly recovery based on the automatic healing mechanism.

[0054] The automatic healing mechanism includes multiple preset recovery strategies, which are attempted sequentially based on the anomaly type or preset priority. As one implementation method, recovery strategies may include: attempting alternative operation paths (i.e., different operation paths for the same function may be pre-stored in the operation intent map, and alternative paths can be attempted); scrolling the screen to find the target element (if it is determined that the target element may be obscured by the screen or not fully loaded), and retrying by returning to the previous state (i.e., simulating a return operation, returning to the interface state of the previous step, and re-executing the current step). If none of the above strategies can recover, manual intervention will be requested. This multi-layered anomaly recovery strategy significantly improves the robustness and success rate of automated tasks in the face of dynamic environmental changes, achieving a technological leap from "passive execution" to "proactive protection."

[0055] In practice, after each execution, the complete execution trajectory data can be uploaded to the learning engine for analysis, thereby identifying steps that frequently trigger fuzzy matching and analyzing the root causes of failure. Then, the corresponding nodes and edges in the operation intent graph are updated to continuously optimize the operation. At the same time, new multimodal operation data recorded by the user can be obtained to correct and train the existing operation intent graph.

[0056] This embodiment provides a method for automatically executing tasks on a mobile device. In response to an automatic task execution command from the mobile device, it acquires a pre-built operation intent map for the target task. This operation intent map is modeled based on multimodal operation data recorded on the mobile device. Real-time interface element structure information is acquired from the current display interface of the mobile device. Based on the operation intent map, the target operation element of the target task is determined from the interface element structure information. The target task is then automatically executed based on the target operation element. By guiding task execution with the pre-built operation intent map, the high latency and high cost of real-time inference are avoided. The real-time acquisition and dynamic matching mechanism of interface element structure information improves the task's adaptability across versions and devices. The combination of these two aspects enhances the efficiency and robustness of mobile task automation.

[0057] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 Step S30 includes steps S301 to S302: Step S301: Obtain the structured operation semantic annotation sequence of the target task based on the operation intent map.

[0058] It should be noted that the structured operation semantic annotation sequence is a set of machine-understandable operation instructions recorded in the operation intent map. Each annotation not only includes the operation type (such as click, swipe), but also the semantic feature description of the target element, and includes precondition assertions and postcondition assertions. Precondition assertions are the key features of the current interface displayed on the screen before the operation occurs (such as the title of the current page, the existence of specific iconic elements).

[0059] Step S302: Based on the structured operation semantic annotation sequence, a multi-level degradation strategy is used to determine the target operation element of the target task in the interface element structure information.

[0060] In practical implementation, when performing adaptive target operation element localization, this embodiment introduces a multi-level degradation strategy to conduct a comprehensive query on the interface element structure information, thereby avoiding the problem that a single matching strategy is prone to failure when facing application version updates, device differences, or dynamic interfaces.

[0061] The multi-level degradation strategy can include a first-level strategy, a second-level fuzzy strategy, and a third-level strategy. The first-level strategy is an exact matching strategy, the second-level strategy is a fuzzy matching strategy, and the third-level strategy is a model localization strategy.

[0062] In one feasible implementation, the multi-level degradation strategy includes: a first-level matching; step S302 may include steps B11~B13: Step B11: Extract the resource identifier, text content, and control class name of the pre-stored target element from the structured operation semantic annotation sequence; It should be noted that the first level of matching is an exact match, which can be performed by using the resource representation, text content and control class name of the target element. Therefore, the resource identifier, text content and control class name of the target element can be extracted from the pre-stored target element in the structured operation semantic annotation sequence.

[0063] Step B12: Use the resource identifier, the text content, and the control class name to perform a first-level match in the interface element structure information; It should be noted that, for example, if the target element recorded in the operation intent map has a unique resource ID and that ID still exists in the current interface, the system directly locks that node. This strategy is the most efficient and has the lowest computational overhead.

[0064] Step B13: When the first-level matching is successful, the node in the interface element structure information that is the same as the resource identifier, the text content, and the control class name is taken as the target operation element of the target task.

[0065] If a node with the same resource identifier, text content, and control class name exists in the interface element structure information, it will be used as the target operation element for the target task. However, if the resource ID is modified in a version update, or if some dynamically generated interface elements do not have a static ID, the first-level matching will fail.

[0066] In specific implementation, if the first-level matching fails, a second-level fuzzy matching is triggered. The second-level matching is relaxed to include text fuzzy matching and regional position constraints. The multi-level degradation strategy includes: second-level fuzzy matching; step S302 may include steps B21~B23: Step B21: If the first-level matching fails, obtain the relative position constraints of the target task; In practice, the second-level matching introduces fuzzy matching and spatial position constraints to handle cases of minor text changes or ID failures. Therefore, the relative position constraints of the target task in the operation intent map can be obtained, that is, the relative positional relationship between the target control and its surrounding reference controls.

[0067] Step B22: Perform a second-level fuzzy matching on the text content, determine the reference element based on the relative position constraint, and filter candidate nodes located within the preset orientation area of ​​the reference element in the interface element structure information; Understandably, for example, suppose the "Agree" button recorded in the intent map becomes "Agree to Agreement" in the current version, direct text matching will fail. In this case, text similarity (such as edit distance or semantic similarity) can be calculated based on the text content. If the similarity exceeds a threshold, it is determined to be a successful fuzzy match. At the same time, in order to prevent false matches, relative position constraints can be used to further filter the confirmed nodes.

[0068] Step B23: Determine the candidate nodes that meet the fuzzy matching conditions and relative position constraints as the target operation elements of the target task.

[0069] In practical implementation, for example, if the operation intent map records that the button is "located to the right of the 'Cancel' button," the system will first locate the "Cancel" button (reference element), and then filter candidate nodes within a preset area to its right. This combination strategy of "fuzzy features + spatial relationships" effectively addresses the positioning challenges brought about by minor adjustments to the interface layout or changes in copywriting, significantly improving the robustness of the solution.

[0070] In practical implementation, if the second-level fuzzy matching fails, the third-level matching can be executed. Therefore, the multi-level degradation strategy also includes: third-level matching; step S302 may include steps B31~B32: Step B31: If the second-level matching fails, capture the screen image of the currently displayed interface on the mobile device; If the second-level matching still fails, the third-level matching is performed. Specifically, the screen image of the currently displayed interface on the mobile device can be captured.

[0071] Step B32: Input the screen image and the semantic description in the structured operation semantic annotation sequence into the visual reasoning model for third-level matching to obtain the target operation elements of the target task.

[0072] When structured information is severely lacking (e.g., the control tree cannot be obtained, or text is in image format) or the interface undergoes drastic changes, the first two levels of attribute- and location-based matching may both fail. In this case, the current screen image is captured and combined with the semantic description in the intent map (e.g., "a red submit button") and input into the visual reasoning model. The model directly predicts the position of the target element at the pixel level by analyzing the correspondence between the image content and the semantic description. The visual reasoning model is a lightweight multimodal model on the device side. The lightweight multimodal model on the device side mainly consists of a lightweight unimodal encoder + a simplified cross-modal connector + a streaming and dynamic fusion backbone network. Thus, it can locate the target operation element of the target task based on the screen image of the current display interface and the semantic description. In the early stage, screen images of different display interfaces and corresponding semantic description sample data of each control can be collected. Then, the model is trained using the sample data to build a lightweight multimodal model.

[0073] This embodiment obtains a structured operation semantic annotation sequence for the target task based on the operation intent graph; according to the structured operation semantic annotation sequence, a multi-level degradation strategy is used to determine the target operation element of the target task from the interface element structure information. Through the multi-level degradation strategy, from exact matching to fuzzy matching to visual reasoning, each level of strategy provides a corresponding solution for specific failure scenarios, thereby achieving adaptive element positioning across devices, versions, and resolutions, and improving the robustness of automatic task execution.

[0074] Based on the first embodiment of this application, in the third embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 Before step S10, the mobile task automatic execution method further includes steps S01 to S03: Step S01: Obtain multimodal operation data recorded on the mobile device.

[0075] It should be noted that multimodal operation data is the raw data used to construct the operation intent graph. In order to fully reproduce the user's operation intent, it is necessary to collect synchronous data from multiple channels. In the initial stage, after the user starts the customized recording service, the screen video stream can be captured through the Android MediaProjection API, and multi-channel data can be collected synchronously to obtain the recorded multimodal operation data.

[0076] In one feasible implementation, step S01 may include steps C11 to C14: Step C11: Obtain the screen video stream, touch trajectory stream, control snapshot stream, environmental status data of the mobile device, and voice input data recorded by the mobile device. Understandably, the screen video stream records visual changes to the interface. It can capture the screen image (30fps) through the AndroidMediaProjection API and listen for interface change events in real time through the onAccessibilityEvent callback of the Accessibility Service to obtain the screen video stream of the user's operation on the mobile interface.

[0077] Each time a user touches the screen, five channels of data are collected simultaneously. Specifically, these include the touch trajectory stream of the touch channel, the spatial snapshot stream of the control channel, the environmental status data of the application channel, the environmental status data of the device channel, and the voice input data of the voice channel.

[0078] The touch trajectory stream records the touch action parameters, namely the coordinates and actions of the finger; the control snapshot stream records the attribute structure of the interface elements of the control below the touch point; and the environment state data records the current foreground package name, page name, screen orientation, input method state, and other context.

[0079] Step C12: Convert the voice input data into voice intent annotation; In practice, the collected real-time user voice input data can be input to the offline speech recognition device, thereby converting the voice input data into text labels, i.e., voice intent annotation, through speech recognition technology.

[0080] Step C13: The screen video stream, the touch trajectory stream, the control snapshot stream, the environmental state data, and the voice intent annotation are frame-level aligned and encapsulated using a unified timestamp protocol to generate enhanced video encapsulation data; Understandably, the collected data is frame-level aligned using a unified timestamp protocol and then encapsulated into enhanced video container data. This multi-channel synchronous encapsulation method ensures the precise temporal correspondence between visual information, touch information, and structured information. The enhanced video container data is data in EVC (Enhanced Video Container) format.

[0081] Step C14: Obtain multimodal operation data based on the enhanced video encapsulation data.

[0082] In practice, the enhanced video encapsulation data can be used as multimodal operation data, and all the data can be precisely aligned to the video frame level, providing a data foundation for subsequent cross-modal alignment.

[0083] Step S02: Perform cross-modal semantic alignment based on the multimodal operation data to obtain a structured operation semantic annotation sequence.

[0084] In practice, the original multimodal data is continuous and redundant, requiring the extraction of meaningful operation fragments and semantic understanding. After receiving the multimodal operation data, the AI ​​learning engine can perform atomic segmentation and cross-modal mapping of operations to obtain a structured sequence of semantically labeled operations.

[0085] In one feasible implementation, step S02 may include steps C21 to C25: Step C21: Calculate the inter-frame difference index between adjacent frames in the multimodal operation data; It should be noted that continuous video streams in multimodal operation data can be segmented. Specifically, the structural similarity index and the color histogram Bach distance between adjacent frames in the video stream can be calculated and used as the inter-frame difference index between adjacent frames.

[0086] Step C22: Detect screen transition points based on the inter-frame difference index; Understandably, if both metrics exceed the preset thresholds simultaneously, the frame is marked as a transition frame, i.e., a screen transition point, corresponding to a functional change in the screen.

[0087] Step C23: Based on the screen transition point, use the video between adjacent transition frames as operation atoms; In practice, the video segments between two adjacent transition frames can be combined into an operation atom, with each operation atom representing an independent interactive action unit, such as "clicking the login button" or "entering an account". This segmentation method based on transition points effectively filters out invalid frames such as hesitation and pauses during user operations and extracts the core operation sequence.

[0088] Step C24: Perform cross-modal alignment of vision and control for each of the described operation atoms to construct a multi-level mapping table; For each operation atom, the engine performs cross-modal alignment from visual pixel coordinates to control nodes, thereby constructing a three-level mapping table for each operation atom, that is, establishing a mapping relationship between visual information and control information, and summarizing to obtain a multi-level mapping table.

[0089] In one feasible implementation, step C24 may include: obtaining the touch hotspot area in the video frame corresponding to the operation atom; obtaining a control tree and touch coordinates based on the multimodal operation data; traversing the leaf nodes in the control tree and filtering nodes whose control attributes and touch coordinates have spatial intersections as candidate control nodes; performing text recognition on the touch hotspot area to obtain text recognition results; when the text attributes of the candidate control node are consistent with the text recognition results, taking the corresponding candidate control node as the target control node and generating a three-level mapping table containing the mapping relationship between visual pixel areas, target control node identifiers, and semantic tags.

[0090] It is understandable that the touch hotspot area refers to the specific location of the user's finger click in the video image and the surrounding visual pixel block. Specifically, the corresponding video frame can be located based on the timestamp of the segmented operation atom, and the touch point coordinates corresponding to the frame can be extracted from the touch trajectory parameters encapsulated in EVC to obtain the touch hotspot area.

[0091] In practical implementation, the complete control tree (AccessibilityNodeInfo tree) of the user operation display interface and the precise physical coordinates of the user's clicks can be obtained based on multimodal operation data. Then, all leaf nodes in the control tree, i.e., the bottom-level interactive actual controls, are traversed to filter out nodes whose control attributes, i.e., the bounds property, intersect with the touch coordinates, and these nodes are marked as "candidate control nodes". Through coarse coordinate positioning, several candidate control nodes that meet the physical conditions can be quickly identified from multiple interface elements.

[0092] In practical implementation, semantics can be extracted from the visual level through OCR text recognition on the device side. For example, if a user clicks a blue button, the OCR will recognize the word "Login" on the blue button image. The text attributes inherent in the candidate control nodes of the text recognition results are compared. When the two match, the candidate node is finally confirmed as the "target control node". After the target control node is locked, a three-level mapping record containing "visual pixel area, target control node identifier, and semantic label" is generated, namely {visual pixel area → control node ID → semantic label}. This is then summarized into a complete three-level mapping table. By constructing the three-level mapping table, we not only know "where was clicked" (visual coordinates), but also "what was clicked" (control ID) and "what it means" (semantic label), thus providing accurate structured input for subsequent intent reasoning.

[0093] Step C25: Generate a structured operation semantic annotation sequence based on the multimodal operation data and the multilevel mapping table.

[0094] It should be noted that multimodal operation data and multi-level mapping tables can be integrated to generate the final structured operation semantic annotation sequence. The structured operation semantic annotation sequence includes the user's operation intent, precondition assertions, and poststate assertions.

[0095] In one feasible implementation, step C25 may include: extracting the front screen state snapshot, touch action parameters, control operation snapshot, and back screen state snapshot of each operation atom from the multimodal operation data; obtaining the voice intent annotation in the multimodal operation data; encapsulating the front screen state snapshot, touch action parameters, control operation snapshot, and back screen state snapshot into quadruple data of operation atoms; inputting the quadruple data, the voice intent annotation, and the multi-level mapping table into a multimodal large language model for intent reasoning, generating a structured operation semantic annotation containing operation intent, preconditional assertion, and poststate assertion.

[0096] It should be noted that after segmenting into multiple operation atoms, the preceding screen state snapshot, touch action parameters, control operation snapshot, and following screen state snapshot of each operation atom can be extracted from the multimodal operation data. The preceding screen state snapshot is the screen interface displayed before the operation occurs. The touch action parameters are the user's touch data, such as click coordinates, swipe trajectory, and long press duration. The control operation snapshot is the information of the specific control being operated, such as the button's ID, text, and position. The following screen state snapshot is the state of the screen interface displayed after the operation occurs.

[0097] By encapsulating the above parameters into quadruple structure data of operation atoms, the quadruple data fully describes the cause and effect of an operation, which facilitates the subsequent understanding of the operation intent by the large language model.

[0098] The Multimodal Large Language Model (LLM) can assemble the quadruple data of each operation atom, the speech intent annotation in the multimodal operation data, and the generated three-level mapping table into a multimodal Prompt, which is then sent to the private multimodal large language model.

[0099] Multimodal Large Language Models (LLMs) combine visual screenshots, control attributes, and user voice intent annotations for deep reasoning. This allows them to identify the purpose of an operation, triggering conditions, and expected results, ultimately outputting structured semantic annotations of the operation, including the operation intent, precondition assertions, and poststate assertions. For example, the model not only recognizes that "the button with ID btn_submit was clicked," but also infers that the intent of the operation is "to submit the form," generating precondition assertions (such as "the submit button is visible and clickable") and poststate assertions (such as "the page redirects to the success page").

[0100] Step S03: Obtain the operation intent map based on the structured operation semantic annotation sequence model.

[0101] In practical implementation, the structured semantic annotations of all operation atoms can be connected in logical order to construct an operation intent graph in the form of a directed acyclic graph. This intent graph not only records the operation steps but also contains rich semantic constraints and state assertions. When multiple successor states are detected, branch edges are automatically generated; when continuous repetition patterns are detected, they are extracted as loop structures; and when the execution does not change as expected, exception handling edges are generated. This makes the task model interpretable and auditable, laying the foundation for subsequent highly robust execution.

[0102] In practical implementation, for multiple EVC recordings of the same task, the engine performs multi-video consensus disambiguation. For each independently generated Operation Intent Graph (OIG) operation edge sequence, a Multiple Sequence Alignment (MSA) algorithm is executed to count the frequency of each operation step: steps appearing in all recordings are marked as core steps (100%), steps appearing in most recordings are marked as optional steps, and steps appearing only once are marked as noise steps and removed, thus significantly improving the robustness of the task model.

[0103] This embodiment acquires multimodal operation data recorded on a mobile device; performs cross-modal semantic alignment based on the multimodal operation data to obtain a structured operation semantic annotation sequence; and models an operation intent graph based on the structured operation semantic annotation sequence. By fusing multimodal data such as visual, structural, action, and voice intent, a large model is used for cross-modal semantic alignment. This multi-verification mechanism of visual + structural + semantic not only accurately locates the target control but also deeply understands the true business intent behind the operation, greatly improving the accuracy and robustness of intent recognition. The operation intent graph (OIG) obtained by modeling based on the structured annotation sequence transforms the originally flat and linear operation data into a three-dimensional decision network containing branches, loops, and exception handling paths. The graph structure not only clearly maps the topological relationship of business flow but also supports the extraction and reuse of complex logic.

[0104] For example, to help understand the implementation flow of the mobile task auto-execution method obtained by combining this embodiment with the above embodiment one, please refer to... Figure 4 , Figure 4 A simplified flowchart of a mobile task auto-execution method is provided, specifically including steps Q1~Q10. Q1 is multi-channel enhanced video acquisition and frame-level synchronization encapsulation, i.e., acquiring multimodal operation data; Q2: frame difference screen jump detection and operation atomic segmentation; Q3: visual-control cross-modal alignment and three-level mapping table construction; Q4: operation intent understanding and semantic annotation based on multimodal LLM; Q5: operation intent graph OIG construction and conditional logic extraction; Q6: multi-video consensus disambiguation and noise filtering; Q7: semantic fingerprint three-level degradation adaptive element localization; Q8: anthropomorphic touch behavior generation and automated operation execution; Q9: during execution, visual loop closure verification and self-healing anomaly recovery are performed. If the verification fails, step Q7 is returned; if the verification succeeds, step Q10 is executed: incremental learning feedback and continuous optimization of the task model, and finally, the model is updated, thereby continuously updating the operation intent graph OIG so that the model can adapt to the iterative updates of the application interface.

[0105] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the mobile task automatic execution method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0106] This application also provides a mobile task auto-execution device, please refer to... Figure 5 The mobile task auto-execution device includes: The acquisition module 10 is used to acquire a pre-built operation intent map of the target task in response to the task auto-execution command of the mobile terminal, wherein the operation intent map is modeled based on multimodal operation data recorded by the mobile terminal.

[0107] The acquisition module 10 is also used to acquire real-time interface element structure information on the current display interface of the mobile terminal.

[0108] The determination module 20 is used to determine the target operation element of the target task in the interface element structure information based on the operation intent map.

[0109] The execution module 30 is used to automatically execute the target task based on the target operation element.

[0110] The mobile task auto-execution device provided in this application, employing the mobile task auto-execution method in the above embodiments, can solve the technical problems of poor execution effect and high cost of current mobile task automation solutions. Compared with the prior art, the beneficial effects of the mobile task auto-execution device provided in this application are the same as those of the mobile task auto-execution method provided in the above embodiments, and other technical features in the mobile task auto-execution device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0111] In one embodiment, the determining module 20 is further configured to obtain a structured operation semantic annotation sequence for the target task based on the operation intent map; Based on the structured operation semantic annotation sequence, a multi-level degradation strategy is used to determine the target operation element of the target task from the interface element structure information.

[0112] In one embodiment, the multi-level degradation strategy includes: a first-level matching; the determining module 20 is further configured to extract the resource identifier, text content, and control class name of the target element pre-stored in the structured operation semantic annotation sequence; The resource identifier, the text content, and the control class name are used to perform a first-level match in the interface element structure information; When the first-level matching is successful, the node in the interface element structure information that is the same as the resource identifier, the text content, and the control class name is taken as the target operation element of the target task.

[0113] In one embodiment, the multi-level degradation strategy further includes: a second-level fuzzy matching; the determining module 20 is further configured to obtain the relative position constraints of the target task when the first-level matching fails. The text content is subjected to a second-level fuzzy matching, and reference elements are determined based on the relative position constraints. Candidate nodes located within the preset orientation area of ​​the reference elements are filtered from the interface element structure information. Candidate nodes that meet the fuzzy matching conditions and relative position constraints are identified as the target operation elements of the target task.

[0114] In one embodiment, the multi-level degradation strategy further includes: a third-level matching; the determining module 20 is also used to capture the screen image of the currently displayed interface of the mobile terminal when the second-level matching fails. The screen image and the semantic description in the structured operation semantic annotation sequence are input into the visual reasoning model for third-level matching to obtain the target operation elements of the target task.

[0115] In one embodiment, the device further includes a construction module, which is used to acquire multimodal operation data recorded by a mobile device; Cross-modal semantic alignment is performed based on the multimodal operation data to obtain a structured operation semantic annotation sequence; An operation intent graph is obtained by modeling the structured operation semantic annotation sequence.

[0116] In one embodiment, the construction module is further configured to calculate an inter-frame difference index between adjacent frames in the multimodal operation data; Detect screen transition points based on the inter-frame difference index; Based on the screen transition point, the video between adjacent transition frames is used as the operation atom; Perform cross-modal alignment of vision and control for each of the aforementioned operation atoms, and construct a multi-level mapping table; A structured operation semantic annotation sequence is generated based on the multimodal operation data and the multilevel mapping table.

[0117] In one embodiment, the construction module is further configured to extract the front screen state snapshot, touch action parameters, control operation snapshot, and back screen state snapshot of each operation atom from the multimodal operation data; Obtain the speech intent annotation from the multimodal operation data; The front screen state snapshot, touch action parameters, control operation snapshot, and rear screen state snapshot are encapsulated into a quadruple of operation atoms; The quadruple data, the speech intent annotation, and the multi-level mapping table are input into a multimodal large language model for intent reasoning, generating a structured operation semantic annotation that includes operation intent, precondition assertion, and poststate assertion.

[0118] In one embodiment, the construction module is further configured to obtain the touch hotspot area in the video frame corresponding to the operation atom; The control tree and touch coordinates are obtained based on the multimodal operation data; Traverse the leaf nodes in the control tree and filter the nodes whose control properties and touch coordinates intersect in space as candidate control nodes; Text recognition is performed on the touch hotspot area to obtain the text recognition result; When the text attribute of the candidate control node matches the result of the text recognition, the corresponding candidate control node is taken as the target control node, and a three-level mapping table containing the mapping relationship between visual pixel regions, target control node identifiers, and semantic tags is generated.

[0119] In one embodiment, the construction module is further configured to acquire screen video streams, touch trajectory streams, control snapshot streams, environmental status data of the mobile device, and voice input data recorded by the mobile device. Convert the voice input data into voice intent annotations; The screen video stream, the touch trajectory stream, the control snapshot stream, the environmental state data, and the voice intent annotation are frame-level aligned and encapsulated using a unified timestamp protocol to generate enhanced video encapsulation data. Multimodal operation data is obtained based on the enhanced video encapsulation data.

[0120] In one embodiment, the execution module 30 is further configured to generate an anthropomorphic touch event containing a Bézier curve trajectory and random speed fluctuations based on the coordinate range of the target operation element. The anthropomorphic touch events are injected into the touch driver layer of the mobile device to automatically execute the target task.

[0121] In one embodiment, the device further includes a recovery module, which is used to acquire screenshot data of operations during the execution of the target task; Perform a closed-loop verification between the screenshot data and the post-state assertions in the operation intent map; When the closed-loop verification result exceeds the preset tolerance, the target task execution is determined to be abnormal, and the automatic healing mechanism is triggered. The target task is restored based on the aforementioned automatic healing mechanism.

[0122] This application provides a mobile task auto-execution device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the mobile task auto-execution method in the above embodiment 1.

[0123] The following is for reference. Figure 6 This document illustrates a structural schematic diagram of a mobile task auto-execution device suitable for implementing embodiments of this application. The mobile task auto-execution device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The mobile task automation device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0124] like Figure 6As shown, the mobile task automation device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in ROM (Read Only Memory) 1002 or a program loaded from storage device 1003 into RAM (Random Access Memory) 1004. RAM 1004 also stores various programs and data required for the operation of the mobile task automation device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, LCDs (Liquid Crystal Displays), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the mobile task automation device to communicate wirelessly or wiredly with other devices to exchange data. While the figures show mobile task automation devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0125] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0126] The mobile task automation device provided in this application, employing the mobile task automation method described in the above embodiments, can solve the technical problems of poor execution effect and high cost of current mobile task automation solutions. Compared with the prior art, the beneficial effects of the mobile task automation device provided in this application are the same as those of the mobile task automation method provided in the above embodiments, and other technical features of this mobile task automation device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0127] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0128] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0129] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the mobile terminal task automatic execution method in the above embodiments.

[0130] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory or Flash Memory), optical fibers, CD-ROM (CD-Read Only Memory), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0131] The aforementioned computer-readable storage medium may be included in a mobile task automation device; or it may exist independently and not assembled into a mobile task automation device.

[0132] The aforementioned computer-readable storage medium carries one or more programs. When the one or more programs are executed by a mobile task auto-execution device, the mobile task auto-execution device causes the mobile task auto-execution device to: respond to a task auto-execution instruction from the mobile terminal, acquire a pre-constructed operation intent map of the target task, wherein the operation intent map is modeled based on multimodal operation data recorded by the mobile terminal; acquire real-time interface element structure information on the current display interface of the mobile terminal; determine the target operation element of the target task based on the operation intent map in the interface element structure information; and automatically execute the target task based on the target operation element.

[0133] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including LAN (Local Area Network) or WAN (Wide Area Network)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0135] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0136] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described mobile task automation method. This solves the technical problems of poor execution performance and high cost in current mobile task automation solutions. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the mobile task automation method provided in the above embodiments, and will not be repeated here.

[0137] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the mobile terminal task automatic execution method described above.

[0138] The computer program product provided in this application can solve the technical problems of poor execution effect and high cost of current mobile task automation solutions. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the mobile task automatic execution method provided in the above embodiments, and will not be repeated here.

[0139] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

[0140] This application discloses A1. A method for automatically executing tasks on a mobile device, the method comprising: In response to the task auto-execution command from the mobile terminal, a pre-constructed operation intent map of the target task is obtained, wherein the operation intent map is modeled based on multimodal operation data recorded by the mobile terminal; Obtain real-time information about the structure of UI elements on the current display screen of the mobile device; Based on the operation intent map, the target operation element of the target task is determined in the interface element structure information; The target task is executed automatically based on the target operation element.

[0141] A2. As described in A1, the step of determining the target operation element of the target task based on the operation intent map in the interface element structure information includes: Based on the operation intent map, a structured operation semantic annotation sequence for the target task is obtained; Based on the structured operation semantic annotation sequence, a multi-level degradation strategy is used to determine the target operation element of the target task from the interface element structure information.

[0142] A3. As described in A2, the multi-level degradation strategy includes: first-level matching; The step of determining the target operation element of the target task from the interface element structure information using a multi-level degradation strategy based on the structured operation semantic annotation sequence includes: Extract the resource identifier, text content, and control class name of the pre-stored target element from the structured operation semantic annotation sequence; The resource identifier, the text content, and the control class name are used to perform a first-level match in the interface element structure information; When the first-level matching is successful, the node in the interface element structure information that is the same as the resource identifier, the text content, and the control class name is taken as the target operation element of the target task.

[0143] A4. As described in A3, the multi-level degradation strategy further includes: a second-level fuzzy matching; After the step of performing a first-level match on the interface element structure information using the resource identifier, the text content, and the control class name, the method further includes: If the first-level matching fails, obtain the relative position constraints of the target task; The text content is subjected to a second-level fuzzy matching, and reference elements are determined based on the relative position constraints. Candidate nodes located within the preset orientation area of ​​the reference elements are filtered from the interface element structure information. Candidate nodes that meet the fuzzy matching conditions and relative position constraints are identified as the target operation elements of the target task.

[0144] A5. As described in A4, the multi-level degradation strategy further includes: third-level matching; After the steps of performing a second-level fuzzy matching on the text content, determining reference elements based on the relative position constraints, and filtering candidate nodes located within the preset orientation area of ​​the reference elements in the interface element structure information, the method further includes: If the second-level matching fails, capture the screen image of the currently displayed interface on the mobile device; The screen image and the semantic description in the structured operation semantic annotation sequence are input into the visual reasoning model for third-level matching to obtain the target operation elements of the target task.

[0145] A6. As described in A1, the steps for pre-constructing the operation intent map include: Acquire multimodal operation data recorded on the mobile device; Cross-modal semantic alignment is performed based on the multimodal operation data to obtain a structured operation semantic annotation sequence; An operation intent graph is obtained by modeling the structured operation semantic annotation sequence.

[0146] A7. As described in A6, the step of performing cross-modal semantic alignment based on the multimodal operation data to obtain a structured operation semantic annotation sequence includes: Calculate the inter-frame difference index between adjacent frames in the multimodal operation data; Detect screen transition points based on the inter-frame difference index; Based on the screen transition point, the video between adjacent transition frames is used as the operation atom; Perform cross-modal alignment of vision and control for each of the aforementioned operation atoms, and construct a multi-level mapping table; A structured operation semantic annotation sequence is generated based on the multimodal operation data and the multilevel mapping table.

[0147] A8. As described in A7, the step of generating a structured operation semantic annotation sequence based on the multimodal operation data and the multi-level mapping table includes: Extract the front screen state snapshot, touch action parameters, control operation snapshot, and back screen state snapshot of each operation atom from the multimodal operation data; Obtain the speech intent annotation from the multimodal operation data; The front screen state snapshot, touch action parameters, control operation snapshot, and rear screen state snapshot are encapsulated into a quadruple of operation atoms; The quadruple data, the speech intent annotation, and the multi-level mapping table are input into a multimodal large language model for intent reasoning, generating a structured operation semantic annotation that includes operation intent, precondition assertion, and poststate assertion.

[0148] A9. As described in A7, the step of performing cross-modal alignment of vision and control on each of the operation atoms and constructing a multi-level mapping table includes: Obtain the touch hotspot area in the video frame corresponding to the operation atom; The control tree and touch coordinates are obtained based on the multimodal operation data; Traverse the leaf nodes in the control tree and filter the nodes whose control properties and touch coordinates intersect in space as candidate control nodes; Text recognition is performed on the touch hotspot area to obtain the text recognition result; When the text attribute of the candidate control node matches the result of the text recognition, the corresponding candidate control node is taken as the target control node, and a three-level mapping table containing the mapping relationship between visual pixel regions, target control node identifiers, and semantic tags is generated.

[0149] A10. As described in A6, the step of acquiring multimodal operation data recorded by the mobile terminal includes: Acquire screen video streams, touch trajectory streams, control snapshot streams, environmental status data of the mobile device, and voice input data recorded by the mobile device; Convert the voice input data into voice intent annotations; The screen video stream, the touch trajectory stream, the control snapshot stream, the environmental state data, and the voice intent annotation are frame-level aligned and encapsulated using a unified timestamp protocol to generate enhanced video encapsulation data. Multimodal operation data is obtained based on the enhanced video encapsulation data.

[0150] A11. The method as described in any one of A1 to A10, wherein the step of automatically executing the target task based on the target operation element comprises: Based on the coordinate range of the target operation element, generate anthropomorphic touch events containing Bézier curve trajectories and random speed fluctuations; The anthropomorphic touch events are injected into the touch driver layer of the mobile device to automatically execute the target task.

[0151] A12. The method as described in any one of A1 to A10, further comprising, after the step of automatically executing the target task based on the target operation element: Obtain screenshot data of operations during the execution of the target task; Perform a closed-loop verification between the screenshot data and the post-state assertions in the operation intent map; When the closed-loop verification result exceeds the preset tolerance, the target task execution is determined to be abnormal, and the automatic healing mechanism is triggered. The target task is restored based on the aforementioned automatic healing mechanism.

[0152] This application also discloses B13. A mobile terminal task auto-execution device, the device comprising: The acquisition module is used to acquire a pre-built operation intent map of the target task in response to the task auto-execution command of the mobile terminal, wherein the operation intent map is obtained by modeling based on multimodal operation data recorded by the mobile terminal; The acquisition module is also used to acquire real-time interface element structure information on the current display interface of the mobile terminal. The determination module is used to determine the target operation element of the target task in the interface element structure information based on the operation intent map; An execution module is used to automatically execute the target task based on the target operation element.

[0153] B14. The apparatus as described in B13, wherein the determining module is further configured to obtain a structured operation semantic annotation sequence for the target task based on the operation intent map; Based on the structured operation semantic annotation sequence, a multi-level degradation strategy is used to determine the target operation element of the target task from the interface element structure information.

[0154] B15. The apparatus as described in B14, wherein the multi-level degradation strategy includes: a first-level matching; the determining module is further configured to extract the resource identifier, text content, and control class name of the target element pre-stored in the structured operation semantic annotation sequence; The resource identifier, the text content, and the control class name are used to perform a first-level match in the interface element structure information; When the first-level matching is successful, the node in the interface element structure information that is the same as the resource identifier, the text content, and the control class name is taken as the target operation element of the target task.

[0155] B16. The apparatus as described in B15, wherein the multi-level degradation strategy further includes: a second-level fuzzy matching; the determining module is further configured to obtain the relative position constraints of the target task when the first-level matching fails. The text content is subjected to a second-level fuzzy matching, and reference elements are determined based on the relative position constraints. Candidate nodes located within the preset orientation area of ​​the reference elements are filtered from the interface element structure information. Candidate nodes that meet the fuzzy matching conditions and relative position constraints are identified as the target operation elements of the target task.

[0156] B17. The apparatus as described in B16, wherein the multi-level degradation strategy further includes: a third-level matching; and the determining module is further configured to capture a screen image of the currently displayed interface of the mobile terminal when the second-level matching fails. The screen image and the semantic description in the structured operation semantic annotation sequence are input into the visual reasoning model for third-level matching to obtain the target operation elements of the target task.

[0157] B18. The apparatus as described in B13, further comprising a construction module for acquiring multimodal operation data recorded by a mobile device; Cross-modal semantic alignment is performed based on the multimodal operation data to obtain a structured operation semantic annotation sequence; An operation intent graph is obtained by modeling the structured operation semantic annotation sequence.

[0158] This application also discloses C19. A mobile task auto-execution device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the mobile task auto-execution method as described above.

[0159] This application also discloses D20. A storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the mobile terminal task automatic execution method as described above.

Claims

1. A method for automatically executing tasks on a mobile device, characterized in that, The method for automatically executing mobile tasks includes: In response to the task auto-execution command from the mobile terminal, a pre-constructed operation intent map of the target task is obtained, wherein the operation intent map is modeled based on multimodal operation data recorded by the mobile terminal; Obtain real-time information about the structure of UI elements on the current display screen of the mobile device; Based on the operation intent map, the target operation element of the target task is determined in the interface element structure information; The target task is executed automatically based on the target operation element.

2. The method as described in claim 1, characterized in that, The step of determining the target operation element of the target task based on the operation intent map in the interface element structure information includes: Based on the operation intent map, a structured operation semantic annotation sequence for the target task is obtained; Based on the structured operation semantic annotation sequence, a multi-level degradation strategy is used to determine the target operation element of the target task from the interface element structure information.

3. The method as described in claim 2, characterized in that, The multi-level degradation strategy includes: first-level matching; The step of determining the target operation element of the target task from the interface element structure information using a multi-level degradation strategy based on the structured operation semantic annotation sequence includes: Extract the resource identifier, text content, and control class name of the pre-stored target element from the structured operation semantic annotation sequence; The resource identifier, the text content, and the control class name are used to perform a first-level match in the interface element structure information; When the first-level matching is successful, the node in the interface element structure information that is the same as the resource identifier, the text content, and the control class name is taken as the target operation element of the target task.

4. The method as described in claim 3, characterized in that, The multi-level degradation strategy also includes: a second-level fuzzy matching; After the step of performing a first-level match on the interface element structure information using the resource identifier, the text content, and the control class name, the method further includes: If the first-level matching fails, obtain the relative position constraints of the target task; The text content is subjected to a second-level fuzzy matching, and reference elements are determined based on the relative position constraints. Candidate nodes located within the preset orientation area of ​​the reference elements are filtered from the interface element structure information. Candidate nodes that meet the fuzzy matching conditions and relative position constraints are identified as the target operation elements of the target task.

5. The method as described in claim 4, characterized in that, The multi-level degradation strategy also includes: a third-level matching; After the steps of performing a second-level fuzzy matching on the text content, determining reference elements based on the relative position constraints, and filtering candidate nodes located within the preset orientation area of ​​the reference elements in the interface element structure information, the method further includes: If the second-level matching fails, capture the screen image of the currently displayed interface on the mobile device; The screen image and the semantic description in the structured operation semantic annotation sequence are input into the visual reasoning model for third-level matching to obtain the target operation elements of the target task.

6. The method as described in claim 1, characterized in that, The steps for pre-building an operation intent map include: Acquire multimodal operation data recorded on the mobile device; Cross-modal semantic alignment is performed based on the multimodal operation data to obtain a structured operation semantic annotation sequence; An operation intent graph is obtained by modeling the structured operation semantic annotation sequence.

7. The method as described in claim 6, characterized in that, The step of performing cross-modal semantic alignment based on the multimodal operation data to obtain a structured operation semantic annotation sequence includes: Calculate the inter-frame difference index between adjacent frames in the multimodal operation data; Detect screen transition points based on the aforementioned inter-frame difference index; Based on the screen transition point, the video between adjacent transition frames is used as the operation atom; Perform cross-modal alignment of vision and control for each of the aforementioned operation atoms, and construct a multi-level mapping table; A structured operation semantic annotation sequence is generated based on the multimodal operation data and the multilevel mapping table.

8. A mobile terminal task automatic execution device, characterized in that, The device includes: The acquisition module is used to acquire a pre-built operation intent map of the target task in response to the task auto-execution command of the mobile terminal, wherein the operation intent map is obtained by modeling based on multimodal operation data recorded by the mobile terminal; The acquisition module is also used to acquire real-time interface element structure information on the current display interface of the mobile terminal. The determination module is used to determine the target operation element of the target task in the interface element structure information based on the operation intent map; An execution module is used to automatically execute the target task based on the target operation element.

9. A mobile terminal task automatic execution device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the mobile task auto-execution method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the mobile terminal task automatic execution method as described in any one of claims 1 to 7.

Images