Operation trajectory construction method, interaction method, device, and storage medium

Abstract

The application relates to the computer technical field and discloses a construction method of an operation track, an interaction method, equipment and a storage medium, which comprises the following steps: acquiring a data set corresponding to an application, the data set comprising a control list corresponding to each page in the application and a specification of the application, the control list being used for recording all the controls in the page which can be executed, and the specification recording at least an operation corresponding to the execution of the controls; determining a user interface transfer graph based on the data set; inputting the user interface transfer graph and the specification into a large language model to obtain an operation track for completing a task. According to the method, the intelligent agent obtained by the method can effectively complete a complex operation, the success rate of the operation is greatly improved, and the user experience is improved.

Description

[0001] This application is a divisional application. The original application has the application number 202411717064.3 and the original application date is November 26, 2024. The entire contents of the original application are incorporated herein by reference. Technical Field

[0002] This application relates to the field of electronic device technology, and in particular to a method for constructing an operation trajectory, an interaction method, a device, and a storage medium. Background Technology

[0003] With the explosive growth of applications installed on electronic devices, users frequently use various apps to complete different tasks in their daily lives and work. However, with the increase in the number of apps and the complexity of their operations, users need to spend a lot of time and energy switching between different apps. This not only reduces the efficiency of users in completing certain functions but also increases the complexity and fatigue of operation.

[0004] To address this issue, mobile phone control agents have emerged. A control agent is an intelligent software tool or service that can simulate user actions and automatically perform tasks in the background. By responding to user commands in real time, the control agent can automatically complete a series of operations on the user's mobile phone, thereby greatly improving efficiency. This automation tool not only reduces the frequency of switching between different applications but also significantly reduces manual operation steps, providing users with a more convenient experience.

[0005] However, as application functions become increasingly complex, many tasks often require more than 20 steps to execute. This operational complexity presents significant challenges to the design and execution of operational agents. In complex workflows, operational agents may encounter various obstacles during execution, such as page loading failures, operation timeouts, or interface changes, leading to execution failures and impacting user experience, thus hindering the improvement of work efficiency. Summary of the Invention

[0006] To address the issue of operational agents easily failing to execute under the aforementioned complex operating environments, this application provides a method for constructing operational trajectories, an interaction method, a device, and a storage medium.

[0007] Firstly, this application provides an interaction method applied to an electronic device, comprising: receiving a first operation from a first user on the electronic device; responding to the first operation, recommending a first intelligent agent to the first user, the first intelligent agent being used to perform a first task; receiving a second operation from the first user on the electronic device; and responding to the second operation, presenting the operation trajectory of the first intelligent agent performing the first task. This facilitates the user's understanding of each step of task execution and allows for replacement or modification as needed.

[0008] As one embodiment of the first aspect, the operation trajectory of the first intelligent agent performing the first task is presented, including: the operation trajectory is presented in the form of natural language text according to the execution order, showing the multiple steps included in the operation trajectory. This allows users to modify it according to their needs.

[0009] As one embodiment of the first aspect, presenting the operation trajectory of the first intelligent agent performing a first task includes: displaying the operation trajectory through a first interface; the first interface further includes at least one control for modifying the operation trajectory. Users can operate on the space as needed, making it more flexible and convenient.

[0010] As an embodiment of the first aspect, modifying the operation trajectory includes at least one of the following: manually editing the operation trajectory; automatically optimizing the operation trajectory; and re-recording the operation trajectory.

[0011] As one embodiment of the first aspect, the second operation is a viewing operation, a selection operation, a confirmation operation, or a modification operation on the first intelligent agent.

[0012] As one embodiment of the first aspect, recommending a first intelligent agent to a first user includes: recommending multiple intelligent agents to the first user in an intelligent agent store, the multiple intelligent agents including the first intelligent agent. This facilitates the user in quickly finding the intelligent agent they need.

[0013] As one embodiment of the first aspect, in response to a first operation, recommending a first intelligent agent to a first user includes: identifying the first user's intent based on the first operation; and recommending the first intelligent agent to the first user according to the intent.

[0014] As one embodiment of the first aspect, the first operation is used to instruct an electronic device to perform a first task.

[0015] As one embodiment of the first aspect, the first operation is to enter a search term in the search box.

[0016] As one embodiment of the first aspect, the first operation is dialogue input in a smart assistant.

[0017] As one embodiment of the first aspect, the first operation is used to receive a message from a second user and recommend a first agent to the user, including: presenting the user with the first agent shared by the second user to the first user via a message.

[0018] As one embodiment of the first aspect, the first task is a multi-user collaborative task, and the message is used to request the first user and the second user to collaborate on the first task based on a first intelligent agent. In a second aspect, this application provides a method for constructing an operation trajectory, including:

[0019] Obtain the dataset corresponding to the application. The dataset includes a list of controls corresponding to multiple pages in the application and the application's instruction manual. The control list records the executable controls on the page, and the instruction manual describes at least the operations corresponding to the executed controls. Determine the user interface transition map based on the dataset. Input the user interface transition map and instruction manual into a large model, such as a large language model, to obtain the operation trajectory used to complete the target task.

[0020] According to the construction method of this application embodiment, since the user interface transfer map can be applied to multiple applications, cross-application operations can be achieved when arranging operation trajectories, no longer limited to a single application. Furthermore, the agent obtained through a large language model can effectively improve the success rate of completing the target task, thus enhancing the user experience.

[0021] As an embodiment of the second aspect of this application, determining the user interface transition map based on a dataset includes: launching an application and obtaining a list of controls corresponding to the current page; operating the controls one by one according to the operation of the controls described in the specification, and recording the control information and operation type corresponding to the current operation; saving the content corresponding to the new page that appears after each operation, obtaining a list of controls corresponding to the new page, and operating the controls in the list of controls corresponding to the new page one by one, until all control lists corresponding to the application are traversed;

[0022] By using the content and control list corresponding to the new page as nodes in the user interface transition graph, and the control information and operation type as edges, the user interface transition graph is obtained. This convenient process ensures that all pages corresponding to the application can be traversed, and that controls are executed effectively.

[0023] As an embodiment of the second aspect of this application, the method further includes: constructing a simulation verification environment to verify all operation trajectories; and retaining the operation trajectory that meets the expected execution requirements when the verified operation trajectory meets the expected values. Verification of the operation trajectory not only improves the accuracy and reliability of operation trajectory generation but also simplifies the operation trajectory creation process, enabling users to quickly and accurately generate and deploy operation trajectories, thereby improving system stability and user experience.

[0024] As an embodiment of the second aspect of this application, all operation trajectories are verified, including: in a simulation environment, based on the description of the operation trajectory, simulating the actual operation scenario, and executing the steps corresponding to the operation trajectory step by step; if each step of the current operation trajectory can be executed according to the preset requirements, then it is determined that the operation trajectory meets the expected execution requirements.

[0025] As an embodiment of the second aspect of this application, the operation trajectory that meets the expected execution requirements is reflowed and input into the large model. By repeatedly reflowing, the quality of the operation trajectory can be improved.

[0026] As an embodiment of the second aspect of this application, optimizing the operation trajectory further includes: acquiring historical operation behavior data for completing the target task and result data for executing the operation trajectory, wherein the result data includes the state generated during the execution of the operation trajectory and the action data during the execution of the operation trajectory; inputting the user's historical operation behavior data and result data into the trajectory optimization model to obtain the optimized operation trajectory. This optimization method results in higher operation trajectory accuracy and can meet the user's personalized needs.

[0027] As an embodiment of the second aspect of this application, the optimized operation trajectory is evaluated. When the operation trajectory completes the target task or completes the preset intermediate steps, the optimized result is labeled with a reward tag; when the operation trajectory contains errors or fails to achieve the expected target task, the optimized result is labeled with a penalty tag. This further improves the quality of the operation trajectory.

[0028] As an embodiment of the second aspect of this application, the state includes: the content displayed on the current page corresponding to the application, the text entered by the user, or the activation state of the control.

[0029] As an embodiment of the second aspect of this application, motion data includes: clicking, swiping, or entering text.

[0030] As an embodiment of the second aspect of this application, the historical operation behavior data includes: the user's historical operation behavior and the historical operation behavior of other users, wherein the similarity between the historical operation behavior of other users and the user's historical operation behavior meets a preset value.

[0031] As an embodiment of the second aspect of this application, the application specification includes at least a summary of the user interface, a description of the functions of the controls, a page navigation diagram, and a task operation sequence.

[0032] Thirdly, this application also discloses an electronic device, including: a memory for storing instructions executed by one or more processors of the electronic device, and a processor for performing the method explained in the first aspect embodiment or performing the method explained in the second aspect embodiment.

[0033] Fourthly, this application also discloses a computer-readable storage medium, characterized in that the computer-readable storage medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the method explained in the first aspect embodiment or the method explained in the second aspect embodiment.

[0034] Fifthly, this application also discloses a computer program product containing instructions, characterized in that, when the computer program product is run on an electronic device, it causes the electronic device to perform the method explained in the first aspect embodiment or the method explained in the second aspect embodiment. Attached Figure Description

[0035] Figure 1A A schematic diagram of a mobile phone interface is shown in some embodiments;

[0036] Figure 1B Schematic diagrams of mobile phone interfaces in other embodiments are shown;

[0037] Figure 2 A flowchart illustrating the method for constructing the operation trajectory provided in an embodiment of this application is shown;

[0038] Figure 3 A schematic diagram of the structure of the electronic device of this application is shown;

[0039] Figure 4A This illustrates the overall architecture for constructing and using the operation trajectory in an embodiment of this application;

[0040] Figure 4B The overall architecture for constructing and using the operation trajectory of another embodiment is shown;

[0041] Figure 4C A flowchart illustrating the construction and use of the operation trajectory in an embodiment of this application is shown;

[0042] Figure 5 A flowchart illustrating a method for constructing an operation trajectory according to an embodiment of this application is shown;

[0043] Figure 6 A flowchart illustrating the automated interactive trajectory traversal process of the UTG according to an embodiment of this application is shown;

[0044] Figure 7 A schematic diagram of the architecture of the application specification of this application is shown;

[0045] Figure 8 A schematic diagram of the user interface transition map corresponding to the application of this application embodiment is shown;

[0046] Figure 9A framework diagram of the agent construction process according to an embodiment of this application is shown;

[0047] Figure 10 A flowchart of the agent optimization method according to an embodiment of this application is shown;

[0048] Figure 11 A flowchart illustrating the estimation and recommendation method for the co-occurrence mechanism according to an embodiment of this application is shown;

[0049] Figure 12 A flowchart illustrating the interaction method according to an embodiment of this application is shown;

[0050] Figure 13 A schematic diagram illustrating the user interface interaction between the user and the mobile phone is provided as an example of an embodiment of this application.

[0051] Figure 14 A schematic diagram illustrating the user interface for interaction with a mobile phone, according to another embodiment of this application, is provided.

[0052] Figure 15 The following is a schematic diagram illustrating the interface of a mobile phone executing an intelligent agent according to an embodiment of this application;

[0053] Figure 16 A block diagram of a SoC according to an embodiment of this application is shown. Detailed Implementation

[0054] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0055] The illustrative embodiments of this application include, but are not limited to, a method for constructing an operation trajectory, an interaction method, a device, a storage medium, and a program product.

[0056] To facilitate understanding of the technical solution of this application, the technical problem to be solved by this application will be explained first.

[0057] As described above, with the increasing number of applications on electronic devices and the growing complexity of their functions, it often requires more than 20 steps to automatically execute tasks by controlling intelligent agents. This makes it easy for the intelligent agents to fail during the execution process, resulting in a reduced user experience.

[0058] To address the aforementioned issues, some embodiments employ instruction recording tools to record actual user operations. This involves recording each step of the user's operation in real-time while the user is using the phone, and then converting these operations into a set of instructions—essentially an agent—that can be automatically executed by the system. The phone can then execute these agents to complete the user-recorded steps and automatically perform specific tasks, thereby replacing manual operation.

[0059] refer to Figure 1A, Figure 1A The diagram shows schematic representations of mobile phone interfaces in some embodiments. For example... Figure 1A As shown, on the phone's interface 10, the user can input the task command "order a coffee" in the input box 11. After selecting the operation to perform, such as selecting the "teaching command" command, the user can manually complete the task of ordering a coffee. The phone's built-in command recording tool will automatically record the user's operation and save the recording. When the user inputs the command "order a coffee" again via voice or other means, the phone will automatically complete the task based on the previously recorded video, fulfilling the user's target task. This method allows users to record various operations, including clicking, swiping, typing text, opening or closing applications, etc., providing users with a simple and intuitive way. The phone can achieve highly personalized automated tasks by mimicking human operating habits. Moreover, this command recording method can capture the user's specific operating habits. No matter how complex or special these operations are, the user only needs to operate according to their usual operating habits, and the command recording tool will automatically record these steps. When the user inputs the set command, the phone will automatically reproduce these operations. In this way, users can record many tasks that can be completed through touch screen operations, without being limited by preset function modules, thereby meeting personalized needs.

[0060] While the aforementioned command recording tools offer advantages in terms of personalization and flexibility, they also have the following drawbacks:

[0061] First, in actual operation, the phone controls the execution of corresponding operations based on the cross-sectional coordinates of the recorded screen. However, different phones have different screen sizes, resolutions, and system versions, which may cause deviations in the execution of the user-recorded operation steps. For example, a click operation recorded on one device may result in a shift in the click position on another device due to different resolutions. For operations requiring high precision, this may lead to the task failing to execute correctly or even triggering erroneous operations, thereby reducing the accuracy and reliability of the intelligent agent's execution.

[0062] Second, when tasks need to span multiple applications and involve complex logical judgments or conditional branching, instruction recording tools cannot solve these problems, for example, Figure 1A As shown, this teaching command approach is limited to learning operations within a single application (APP). Users cannot create cross-application operations during the recording process, nor can they handle scenarios requiring multiple interactions or complex judgments, greatly reducing its practicality.

[0063] Third, for tasks with lengthy or repetitive processes, even slight user oversights during recording can reduce accuracy, resulting in an AI agent that fails to accurately reflect the complexities of the operation. Therefore, users must carefully execute each step to ensure recording accuracy and prevent the generated AI agent from malfunctioning. This necessitates high concentration and patience during recording. When modifications or adjustments are needed to a pre-recorded AI agent, the entire process often needs to be re-recorded. This repetitive work further increases the user's workload and reduces operational efficiency.

[0064] In other embodiments, users can create intelligent agents using function orchestration tools on electronic devices, such as the Huawei Skill Center. These orchestration tools allow users to create a complete task flow to obtain an intelligent agent by dragging and combining different function blocks or operation steps. These elements can include built-in system operations, such as opening an application, setting an alarm, or sending a message; or services provided by third-party applications, such as opening an application at a specific time or performing a series of operations when a certain event is triggered. By combining these function blocks in a preset logical manner, users can customize and create their own intelligent agents according to their needs and preferences, achieving automated operation of the mobile phone.

[0065] refer to Figure 1B , Figure 1B Illustrations of mobile phone interfaces in other embodiments are shown. For example... Figure 1B As shown, the mobile interface 20 has fixed statements, such as "When...execute...", and fixed functional modules. For example, in the scene card under the "When" statement, the user can input a task such as "Good Morning Broadcast" or a specific time such as "7:30 AM on a legal working day (skipping legal holidays)". Under the "Execute" statement, the user can select fixed functional blocks, such as "Connect Mobile Phone", "Designate Speaker", "Play the following content", etc. These functional modules can also include sub-modules. For example, under "Designate Speaker", there are specific functional blocks such as "...play...wake-up ringtone". The user can select the time, such as 3 seconds, and select the song name "Canon" in the editable part of the corresponding statement to complete the editing of the operation trajectory. This arrangement tool uses fixed modules. Users only need to drag and arrange functional blocks on the visual interface to complete the creation of tasks without writing code. This allows ordinary users to easily create automated tasks and obtain edited intelligent agents.

[0066] However, while function orchestration tools offer users convenience in creating automated tasks, handling complex tasks, such as those spanning multiple applications, involves more intricate programming statements. For example, the logical relationships between function blocks, the use of conditional statements, and the setup of loops and branches can still be challenging for users without a programming or technical background to understand and use these tools. Furthermore, for complex agents, adding, deleting, or modifying a step may require users to re-examine the entire orchestration trajectory; otherwise, a slight oversight can lead to the agent failing to execute correctly or failing after completing a portion of the steps, thus failing to complete the target task. Moreover, in this orchestration method, the number and types of function blocks involved are predefined by the system or third-party applications, limiting the user's orchestration to these scenarios and failing to meet the user's need for more personalized agents.

[0067] To address the issues mentioned above regarding the limitations of automated operations using instruction recording methods, such as low recording accuracy, inability to execute complex operations across applications, and the potential failure of generated agents to execute correctly due to minor errors in user recording, as well as the high requirements for user orchestration capabilities, limited functional modules, and the tendency for complex agents to fail, this application provides a method for constructing operation trajectories. This method constructs operation trajectories by obtaining the application's user interface transition graph and a large model. The operation trajectory records and describes the sequence of manipulations or operational steps required to complete a target task. These sequences or steps can be executed by an agent, meaning the agent can execute these operation trajectories to complete the user's target task. The user interface transition graph is a graphical model used to represent the states of the user interface and the transition relationships between interfaces. Each state represents a specific configuration or view on the user interface, and a transition represents the user's action of moving from one state to another.

[0068] Since the user interface transition graph can cover all applications and their corresponding functions, it can be used to obtain the operation trajectories for all tasks, which are then executed by agents. In the following text, an agent refers to an entity that completes a target task based on an operation trajectory. These agents can be pre-generated and stored locally or downloaded online. When a user inputs a command, these agents corresponding to the command are directly retrieved, and the target task corresponding to the command is completed based on the agents. This method of constructing operation trajectories does not impose requirements on user actions, is not limited by functional modules, and can also achieve cross-application operations, effectively completing complex operations. Furthermore, the success rate of execution of operation trajectories (agents) generated through a large model is greatly improved, enhancing the user experience.

[0069] In some embodiments of this application, the large model can be a large language model, a multimodal model, a speech model, etc. The following embodiments will use a large language model for illustration.

[0070] The method for constructing the operation trajectory according to the embodiments of this application will be described below with reference to the accompanying drawings.

[0071] refer to Figure 2 , Figure 2 This diagram illustrates a flowchart of a method for constructing an operation trajectory according to an embodiment of this application. This method can be used in electronic devices, such as mobile phones, computers, servers, and other devices with computing capabilities. Figure 2 As shown, the method includes, but is not limited to, the following steps:

[0072] S210, Obtain the dataset corresponding to the application. The dataset includes a list of controls corresponding to multiple pages in the application and the application's instruction manual.

[0073] This step is the data construction process, mainly involving various data related to the target application, such as lists of controls corresponding to multiple pages in the application and the application's manual, to form a dataset.

[0074] Each application contains many different pages, and each page corresponds to a list of controls. The control list records all executable controls on the corresponding page—the basic elements for user interaction with the application. These elements help users perform specific operations or input information. The phone can identify and understand all executable elements on each page based on the control list. Control types include, but are not limited to: buttons, text fields, switches, sliders, dropdown lists, progress bars, labels, images, links, list views, and navigation bars.

[0075] The application's documentation at least describes the operations corresponding to the controls, such as clicking a button, entering text, or switching options. In some embodiments, the documentation also includes a summary of the UI interface, detailed descriptions of control functions, a diagram of page transitions, and operation sequences for common tasks. This documentation provides additional contextual support for the interaction trajectories generated by the UITransition Graph (UTG), enabling the system to have a more comprehensive and in-depth understanding of the app's functionality.

[0076] S220, determine the user interface transition map based on the dataset.

[0077] A user interface transition graph is a graphical structure used to represent the transition relationships between different user interfaces within an application. It uses nodes to represent different states of the application (e.g., pages, windows) and edges to represent user action paths (e.g., clicks, swipes) between interfaces. It can be used to analyze and generate interaction trajectories within an application, helping automation tools understand and simulate user operation flows.

[0078] In some embodiments, the application's corresponding interface and the list of controls corresponding to the interface can be used as nodes in the user interface transition graph. The operations performed on each control, such as action type and control information, are used as edges in the user interface transition graph to form the user interface transition graph. The action type may include, but is not limited to, clicking a button, entering text, or toggling an option. The control information includes, but is not limited to, the control's name, location, and operation description.

[0079] S230 inputs the user interface transfer map and instruction manual into the large language model to obtain the operation trajectory used to complete the target task.

[0080] Large Language Models (LLMs) are pre-trained models capable of generating operational trajectories. Their training sets are derived from application datasets and instruction manuals within the broader context of big data. LLMs utilize input user interface transfer graphs and instruction manuals to automatically generate intelligent agents (operational trajectories) with task logic and operational steps. These agents can be automatically executed by electronic devices upon receiving user commands, thus automating the user's target task.

[0081] Therefore, according to the operation trajectory construction method of this application embodiment, since the user interface transfer map can be applied to all applications, cross-application operations can be realized when arranging operation trajectories, no longer limited to a single application. Furthermore, the agent obtained through a large language model can effectively improve the success rate of completing the target task, enhance the user experience, and facilitate the application and promotion of this function.

[0082] It should be noted that the operational agent or intelligent agent in the embodiments of this application can be understood as an entity corresponding to an operation trajectory for completing a certain task. That is, the operation trajectory in this application can be understood as an intelligent agent that assists the user in completing the target task.

[0083] It is understood that the method for constructing the operation trajectory provided in this application can be executed by an electronic device, including but not limited to mobile phones, smart TVs, wearable devices, tablets, desktop computers, computers, wearable devices (e.g., smartwatches, smart glasses, helmets), virtual reality (VR) devices, augmented reality (AR) devices, wireless devices in industrial control, wireless devices in self-driving, wireless devices in smart grids, wireless devices in transportation safety, wireless devices in smart cities, wireless devices in smart homes, servers, and so on.

[0084] Understandably, for ease of description, the following text will refer to electronic devices as mobile phones.

[0085] Furthermore, this application does not limit the application scenarios of intelligent agents and their related agent stores. They can be widely applied to the following scenarios:

[0086] Daily life automation scenario: In this scenario, users can set up smart agents through their mobile phones to automatically complete daily tasks, such as automatically turning off airplane mode, playing music, checking the weather, and sending schedule reminders every morning.

[0087] Workflow optimization scenarios: In work scenarios, users can leverage intelligent agents to automate tasks such as handling emails, scheduling meetings, organizing work logs, or managing projects. For example, an agent can be set to automatically archive or mark specific emails as pending after receiving them, or automatically open a meeting application and remind the user at a certain time. Through these automated processes, users can focus on more creative and complex tasks, thereby improving work efficiency.

[0088] Smart Home Control Scenarios: The intelligent agent constructed using the method described in this application can be applied to scenarios involving the coordinated control of smart home devices. For example, a user can set up an intelligent agent to automatically turn on the air conditioner and water heater before returning home from get off work, and adjust the lighting based on location and time. Through deep integration with smart home devices, users can enjoy a seamless smart living experience.

[0089] Cross-application task integration scenarios: Users can create and download smart agents that integrate functions from multiple applications through the smart agent store on their mobile phones. For example, a smart agent can be created to simultaneously handle price comparison, order placement, and logistics tracking for online shopping, thereby reducing manual steps.

[0090] The hardware structure of the electronic device will be introduced first.

[0091] The following describes the method for constructing the interaction trajectory in the embodiments of this application, taking into account the specific structure of the electronic device.

[0092] refer to Figure 3 , Figure 3 A schematic diagram of the structure of the electronic device 100 of this application is shown.

[0093] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) connector 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0094] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0095] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0096] The processor 110 can generate operation control signals based on the instruction opcode and timing signals to control the instruction fetching and execution.

[0097] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0098] In some embodiments, the processor 110 acquires a dataset corresponding to the application. The dataset includes a list of controls for each page in the application and a specification for the application. The control list records the executable controls on the page, and the specification at least describes the operations corresponding to the executed controls. The processor 110 can determine a user interface transition graph based on the dataset and input the user interface transition graph and specification into a large language model to obtain an operation trajectory, i.e., an intelligent agent, for completing the task.

[0099] In some embodiments, the processor 110 can verify the agent. That is, in a simulated environment, the processor 110 simulates actual user operation scenarios, verifying each step of the agent's operation as expected, and ensuring its stable and correct execution under various possible usage scenarios. Verification not only identifies potential problems with the agent in practical applications, facilitating timely correction, but also provides data support for agent optimization. Automated verification improves the accuracy and reliability of agent generation, eliminating the need for manual user adjustments, enabling users to quickly and accurately generate and deploy agents, thus improving system stability and user experience.

[0100] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0101] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0102] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0103] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0104] In some embodiments, both the mobile communication module 150 and the wireless communication module 160 can enable the electronic device 100 to communicate with the cloud. The processor 110 obtains a dataset from the cloud, processes the dataset to obtain a user interface transition map, and finally obtains an intelligent agent.

[0105] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0106] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0107] In some embodiments, the display screen 194 may display the labels of these agents, or the agents corresponding to various steps, and display the corresponding operations and interfaces when the agents are executed.

[0108] To facilitate understanding of the method for constructing and applying the operation trajectory in the embodiments of this application, the overall architecture diagram for constructing and using the operation trajectory in the embodiments of this application is first described below. Here, the operation trajectory can be understood as an intelligent agent.

[0109] refer to Figure 4A , Figure 4A The overall architecture for constructing and using the operation trajectory in an embodiment of this application is shown. For example... Figure 4AAs shown, the architecture includes a UTG generation and analysis module 411, an agent generation and editing module 412, an automated agent verification and production module 413, a reinforcement learning trajectory optimization module 414, an intelligent recommendation engine 415, an agent store 421 (AgentStore), and a user interaction and feedback module 422. Among these modules, the UTG generation and analysis module 411, the agent generation and editing module 412, the automated agent verification and production module 413, the reinforcement learning trajectory optimization module 414, and the intelligent recommendation engine 415 can be located on a cloud-based device 410 for building and recommending agents. The agent store 421 (Agent Store) and the user interaction and feedback module 422 can be located on a user terminal device 420 for receiving user commands, displaying the corresponding agent, and executing the agent.

[0110] In some embodiments, such as Figure 4B The overall architecture for constructing and using the operation trajectory shown in another embodiment is such that the automated agent verification production platform 430, corresponding to the automated agent verification production module 413, is a separate device, and the other modules of this structure are integrated with... Figure 4A The module layout is the same.

[0111] In other embodiments, these modules can be installed in the user terminal device to complete the construction of the intelligent agent and its interaction with the user. This application does not limit the device to which these modules belong.

[0112] The following is combined with Figure 4A And refer to Figure 4C The functions of each module are described separately. Figure 4C A flowchart illustrating the construction and use of the operation trajectory in an embodiment of this application is shown.

[0113] like Figure 4A and Figure 4C As shown, the UTG generation and analysis module 411 can also be understood as... Figure 4B The app auto-traversal tool in the software generates a complete UTG (User-Generated Trace) by traversing the app's user interface, recording potential operation paths. Analyzing the UTG provides users with basic interaction trajectories, reducing the complexity of creating intelligent agents and lowering the operational threshold. This traversal tool can traverse all applications and obtain their UTGs, enabling cross-application operations and accurately generating interaction trajectories even for complex operations. The processing implemented by this module corresponds to... Figure 4C In step ①, the APP automatic traversal tool traverses the APP to obtain the user interface transition map.

[0114] The agent generation and editing module 412 is used to create and edit custom agents based on the interaction trajectories provided by UTG. It can visualize the steps of agent generation and display them on the interface. Users can customize the agent through screen recording or adjust and modify each step of the agent to better suit customer needs. The processing implemented by this module corresponds to... Figure 4C Step ② in the process is to construct the trajectory.

[0115] Automated intelligent agent verification production module 413 or ( Figure 4B The automated agent verification and production platform (430) is used to automate the functional and stability verification of agents. After the agent is generated, the platform comprehensively verifies the agent through simulated environments and various test scenarios to ensure stable operation under various conditions. This platform reduces the cost and errors of manual testing through an automated verification process, thereby improving the reliability of the agent. The processing implemented in this module corresponds to... Figure 4C Step ③ in the process is automated verification.

[0116] The reinforcement learning trajectory optimization module 414 is used to optimize the agent's operational trajectory by analyzing user data on the agent's usage and employing reinforcement learning algorithms. Through continuous learning and adjustment, the system's execution efficiency and adaptability to the agent can be improved, enabling the agent to better meet the user's actual needs and provide more efficient automated services in complex scenarios. The processing implemented by this module corresponds to... Figure 4C Step 4 in the process.

[0117] The intelligent recommendation engine 415 analyzes user operating habits, historical data, and preference information to provide personalized agent recommendations. Based on a co-occurrence mechanism, this engine can recommend agents related to the user's frequently used operations or display other similar agents commonly used by the user. This helps users quickly find agents that suit their needs. The processing implemented in this module corresponds to... Figure 4B Step 5 in the process is intelligent recommendation.

[0118] The Agent Store 421 is the core of the entire system, providing users with a platform to share and download agents. Users can not only download agents created by other users from the Agent Store, but also upload their own creations for others to use and evaluate. The Agent Store promotes communication and innovation among users, enabling the entire ecosystem to continuously evolve and enrich.

[0119] The user interaction and feedback module 422 collects user feedback during the use of the intelligent agent and provides an interaction interface between the user and the system. Through continuous user feedback, the system can continuously improve the design of the intelligent agent and the recommendation mechanism, thereby improving user satisfaction and the overall performance of the system. The processing implemented by this module corresponds to... Figure 4B Step 6 in the process involves user editing.

[0120] The method for constructing operation trajectories provided in this application will be described in detail below with reference to the accompanying drawings. This method can be applied to applications such as... Figure 3 In electronic devices with the hardware structure shown, or applied to devices having such... Figure 4A The software structure shown is used in electronic devices. These hardware or software structures may have more or fewer components than those shown, or combine or separate certain components, or have different component arrangements, etc., in electronic devices with similar hardware and software structures.

[0121] refer to Figure 5 , Figure 5 A flowchart illustrating a method for constructing an operation trajectory according to an embodiment of this application is shown. This method can be executed by a mobile phone, such as... Figure 5 As shown, the method for constructing the operation trajectory includes S510-S540.

[0122] S510, Data Construction.

[0123] This process involves data collection and organization. By collecting and organizing various types of data related to the target app, a complete production dataset is created. Data construction includes an automated interaction trajectory traversal process based on the User Interface Transition Graph (UTG) and comprehensive data from the app's documentation.

[0124] The automated interactive trajectory traversal process of the UTG in this application embodiment and the APP manual will be described in detail below.

[0125] First, the automated interactive trajectory traversal process of UTG will be explained.

[0126] In some embodiments, the automated interaction trajectory traversal process of UTG can also be understood as the traversal process of the control list corresponding to the page of the application.

[0127] refer to Figure 6 , Figure 6 A flowchart illustrating the automated interactive trajectory traversal process of a UTG according to an embodiment of this application is shown. Figure 6 As shown, the automated interactive trajectory traversal process includes S511-S516.

[0128] S511, open the application.

[0129] The application can be opened automatically based on user instructions, or it can be opened manually by the user. This application does not limit this method.

[0130] S512, Get the list of page controls.

[0131] By obtaining a list of controls, we can ensure that the phone can recognize and understand all the elements on each page, which can be executed. For example, when the user clicks the share function (element), the phone performs the click operation and opens the next screen.

[0132] After obtaining the list of controls, each control is operated on one by one using a breadth-first or depth-first strategy. For example, clicking a button, entering text, or toggling options.

[0133] S513, determine if the list is empty.

[0134] If the list is empty, it means that all controls in the list have been traversed, so execute S516 to end the traversal.

[0135] If the list is not empty, execute S514, click the control, and remove it from the list. Remove the controls that have been processed from the list, and then continue editing the other controls in the list until the list is empty, at which point the editing is complete.

[0136] S515 determines whether the page has changed.

[0137] We determine whether a new page has been entered by observing changes in the page. If the page changes, it indicates that a new page has been entered. At this point, we need to continue to retrieve the current page control list, return to S512, and traverse the controls on the current page until the page no longer changes.

[0138] If the page does not change, then execute S516.

[0139] S516, traversal complete.

[0140] During the traversal of the aforementioned operation trajectory, for each operation on a control, the system records the control information and operation type involved in the current operation, and takes a screenshot of the newly appeared page to construct data. This data will be used to construct the UTG in the subsequent trajectory generation stage. The state after each operation, such as screenshots and control lists, is considered a node in the UTG, while each operation, such as action type and control information, forms an edge in the UTG.

[0141] Using the above convenient method, each application's corresponding page and its controls can be traversed to obtain the executable operation trajectory.

[0142] In some embodiments, the dataset may also include user-recorded videos. For example, when a user completes a target task, they may record a video of themselves completing the task via screen recording. This video serves as the basic operation trajectory and does not require UTG orchestration, allowing for a faster acquisition of the intelligent agent.

[0143] The following is a further explanation of the application's instruction manual.

[0144] refer to Figure 7 , Figure 7 A schematic diagram of the application specification of this application is shown. For example... Figure 7 As shown, the construction process of the application manual mainly includes three parts: data collection, annotation, and application manual construction. Operation data collection can be achieved through several methods: user-driven exploration based on page traversal, such as clicks and swipes; large-scale model-assisted exploration based on task samples, i.e., obtaining corresponding operations based on big data; operations obtained during the execution of the application's intelligent agent; and manual collection, i.e., user demonstrations of trajectories and free user operations. This collected data is obtained by the event listening module and reported to the UI operation module, resulting in the corresponding operation sequence. Simultaneously, screenshots of the corresponding application interface and structured UI representations form interface screenshot and structured representation data pairs, clarifying and detailing the functions, behaviors, and interface performance of each interaction module. This data ultimately generates the UI interface summary, control function descriptions, page jump relationship diagrams, and task operation sequences in the manual. The content recorded in the manual provides additional contextual support for the operation trajectories generated by UTG, enabling the system to have a more comprehensive and in-depth understanding of the APP's functions, thus improving the accuracy of the generated operation trajectories.

[0145] Now back Figure 5 ,right Figure 5 The other steps in the process will be explained further.

[0146] S520 determines the user interface transition map based on the constructed data.

[0147] In some embodiments, the application's corresponding interface and the list of controls corresponding to the interface can be used as nodes in the user interface transition graph. The operations performed on each control, such as the action type and control information, are used as edges in the user interface transition graph to form the user interface transition graph.

[0148] refer to Figure 8 , Figure 8 A schematic diagram of the user interface transition map corresponding to the application of this application embodiment is shown. For example... Figure 8As shown, each application corresponds to many application interfaces. The content corresponding to these interfaces is used as nodes, where the interface content can be in the form of screenshots. Operations on elements or functional modules within the interface are used as edges to draw the user interface transition graph for the entire application. From the user interface graph, all operation trajectories corresponding to the application can be obtained, for example, as... Figure 8 As shown, the mobile phone desktop interface is used as the starting point of the operation trajectory. After passing through step 1, intermediate interface 1, step 2, intermediate interface 2, and step 3, the user's target task is finally completed, namely, opening a function interface of an application. This interface is used as the endpoint of the operation trajectory. The starting interface screenshot, intermediate interface screenshot, and endpoint interface are combined through corresponding steps 1, 2, and 3 to obtain the operation trajectory (intelligent agent) corresponding to the task target. This method of obtaining the operation trajectory not only systematically records all possible interaction paths of the user interface but also provides rich data support for generating and verifying intelligent agents. Furthermore, for each application, all its corresponding interaction paths are traversed, meaning that any task within the application can form an intelligent agent, comprehensively covering all possible operation paths, providing users with a more convenient and faster operating experience, increasing user dependence, and facilitating widespread adoption.

[0149] S530 inputs the user interface transfer graph and instruction manual into the Big Prophecy model to obtain the intelligent agent.

[0150] refer to Figure 9 , Figure 9 A framework diagram of the agent construction process according to an embodiment of this application is shown. Figure 9 As shown, the UTG trajectory and the app's instruction manual are used as datasets and input into a Large Language Model (LLM). The LLM can combine these datasets to understand and generate natural language text, expressing the operation trajectory for completing the target task step by step in text form. This text can be executed by the mobile phone or viewed by the user, thus creating an intelligent agent. This intelligent agent is unverified and can also be called a pseudo-intelligent agent. Compared to intelligent agents in existing technologies, this type of agent has higher accuracy, effectively increasing the probability of successful task execution and improving the user experience.

[0151] In some embodiments, a video recorded by the user completing a target task can be input into a large language model. The large language model can understand the content of the video and generate text, which can be executed by the mobile phone or viewed by the user to obtain an intelligent agent. This recording method can obtain the intelligent agent desired by the user more quickly.

[0152] In some embodiments, pseudo-intelligent agents can be verified in order to further improve the correctness and reliability of the intelligent agent.

[0153] Continue to refer to Figure 5 and Figure 9 Step S540: Verify the intelligent agent.

[0154] In a simulated environment, the mobile phone system, based on the agent's description, simulates the user's actual operating scenarios, automatically verifying each step of the agent's operation to ensure its correctness and reliability. This ensures the agent can stably and accurately complete tasks under various possible usage scenarios. Furthermore, the verification of the agent identifies potential problems in real-world applications, allowing for timely resolution and adjustments to ensure each agent can be executed effectively to achieve the user's target tasks.

[0155] like Figure 5 and Figure 9 As shown, to further enhance the accuracy and diversity of agents, validated data, such as successfully validated agents, is fed back into the dataset. Through continuous iteration via this data feedback mechanism, the large language model is optimized, thereby improving its accuracy. This gradually enhances the system's intelligence level in generating and validating agents, ensuring higher reliability and accuracy. Furthermore, the resulting agents can be repeatedly validated to ensure the stability of the selected agents' functionality.

[0156] In some embodiments, such as Figure 9 As shown, the dataset can also include user logs. These logs can include, but are not limited to, data such as user operating habits, the intelligent agent actually used by the user, or the user's operational trajectory. Integrating user logs into the production and training of the large language model allows for the creation of intelligent agents that better adapt to users' personalized needs based on the large oracle model, further improving the user experience.

[0157] In some embodiments of this application, in order to further optimize the execution efficiency and adaptability of the agent, the agent is iteratively optimized using a trajectory optimization model.

[0158] refer to Figure 10 , Figure 10 A flowchart illustrating an embodiment of the intelligent agent optimization method of this application is shown. This optimization can be performed by a mobile phone or a cloud server. Figure 10 As shown, the agent optimization method includes S1010-S1060.

[0159] S1010: Input the operation trajectory into the trajectory optimization model.

[0160] In the embodiments of this application, the trajectory optimization model is trained based on data such as user actions, the execution results of the agent, and the environmental state corresponding to the agent. User actions include, but are not limited to, swiping, clicking, the order of operations, and selecting a path to perform the task. The execution results of the agent include, but are not limited to, information such as whether the agent succeeded or failed, the percentage of completion, or a loading failure at a certain step.

[0161] In some embodiments, the trajectory optimization model may be a Bézier curve fitting model or a polynomial curve fitting model, etc. The optimization algorithm may be gradient descent, Kalman filter, or genetic algorithm, etc., and the embodiments of this application are not limited thereto.

[0162] It should be noted that in the embodiments of this application, in order to protect the privacy and security of users, sensitive user information will be de-identified during the collection process to ensure the security of user identity information.

[0163] S1020 outputs the optimized trajectory.

[0164] The optimized operation trajectory can be used directly, for example, it can be stored in the cloud for users to download and use when needed, i.e., intelligent agent deployment. Simultaneously, personalized data of users using the intelligent agent is collected, such as user operation habits and usage frequency. Based on this data, the trajectory optimization model is trained, and the intelligent agent obtained based on the trajectory optimization model can better meet users' personalized needs.

[0165] S1030, obtain the return data of current network users.

[0166] Among them, live user feedback data refers to data collected from the actual usage environment, which can include historical operational behavior data of users completing target tasks. Because this data reflects user behavior and feedback in real-world scenarios, it is crucial for the continuous optimization and improvement of the model. Live user feedback data can include user behavior data, such as clicks, swipes, or input; user feedback data, such as direct evaluations and ratings of the agent, complaints, and suggestions; system log data, such as records of user habitual operations, and which type of agent is used more frequently. In addition, it can also include result data of the execution trajectory, such as whether the agent succeeded or failed, the percentage of completion, or information about the status of a certain step.

[0167] S1040, Data Quality Filtering. This determines whether the data quality meets the required standards.

[0168] When the data meets the requirements, such as when a user uses an agent to share a contact using two different trajectories, and the agent is used 80% of the time or executes successfully, the data is considered acceptable. Then, S1050 is executed, and these data are used as training data to perform reinforcement learning training on the trajectory optimization model. This allows the optimized agent to perform more intelligently and efficiently in different environments and tasks, while improving the user's operating experience and efficiency.

[0169] If the usage rate of an agent is lower than a set value, such as 10%, or if the agent fails to execute, it will be eliminated, and S1060 will be executed to discard low-quality data.

[0170] Therefore, by further optimizing the intelligent agent, its execution efficiency and adaptability can be improved, thereby enhancing the user experience.

[0171] In one embodiment of this application, during the optimization process, the execution process of the intelligent agent can be abstracted into a state space. In this state space, each state represents the specific environmental situation or execution process of the agent at a certain moment. For example, a state can represent the content displayed on the current page, the text entered by the user, the activation state of a control, etc. By mapping the complex execution process to a state space, the system can more accurately understand and analyze the performance of the intelligent agent in different contexts.

[0172] In each defined state, the agent needs to select an optimal action (action data) to execute in order to achieve the desired effect. This action can be a common interactive operation such as clicking, swiping, or entering text. The process of action selection not only affects the current execution effect but also has a chain reaction on the agent's subsequent behavior. Therefore, choosing the appropriate action is crucial.

[0173] Furthermore, during the agent's optimization process, to guide the agent towards the optimal direction, the system defines a corresponding reward mechanism for each action. The definition of the reward is determined by evaluating the agent's actual performance during execution. For example, when the agent successfully completes a task or achieves an intermediate goal, the system provides a positive reward, such as a positive label or a higher score (e.g., 8 points or above). Conversely, when the agent makes an error or fails to achieve the expected result, a penalty is imposed, such as a negative label or a lower score (e.g., below 5 points). By introducing a reward mechanism, the agent can continuously optimize its behavioral strategy through reinforcement learning, gradually learning the optimal path. For example, during training, the trajectory optimization model continuously adjusts the agent's behavioral strategy based on the defined reward information, gradually guiding it towards the optimal path.

[0174] The intelligent agent optimization method according to the embodiments of this application, through the design of data feedback, state definition, action selection, and reward mechanism, combined with repeated reinforcement learning training, enables the optimized intelligent agent to better adapt to various complex operation scenarios and provide users with a more intelligent and personalized service experience.

[0175] In one embodiment of this application, to further enhance the user experience, a co-occurrence mechanism is used to obtain the user's personalized preferences, and intelligent agents that meet the user's personalized needs are recommended. For example, by collecting the user's own log data, or log data of similar users, the user's personalized preferences are analyzed, and based on these preferences, intelligent agents that meet the user's personalized needs are recommended.

[0176] refer to Figure 11 , Figure 11 A flowchart illustrating the estimation and recommendation method for the co-occurrence mechanism according to an embodiment of this application is shown. Figure 11 As shown, the recommended scheme employs two calculation methods: the first is a high-frequency operation agent recommendation algorithm, which can be executed by the high-frequency operation agent recommendation module; the second is a similar user agent recommendation algorithm.

[0177] The first type, the recommendation of high-frequency operating agents, includes steps S1111-S1113.

[0178] S1111, User log data return.

[0179] In the high-frequency operation intelligent agent recommendation module, the mobile phone system first performs data feedback on the user's historical operation data. This feedback data includes, but is not limited to, frequently used operations, frequently used functions, and daily task processing flows in different mobile applications.

[0180] In the embodiments of this application, to ensure the security of user personal information and protect user privacy, strict anonymization processing is performed on the data at each stage during the data return process. This involves removing information directly associated with the user's identity while retaining necessary operational data and behavioral patterns. The set of these anonymized data can be represented as D = {d1, d2, ..., d...} n}, where each d i It represents a specific operational behavior data record.

[0181] S1112, Behavioral Mining.

[0182] In some embodiments, in-depth behavioral mining is performed on the anonymized data to analyze user operating habits and identify high-frequency operation patterns. Data mining algorithms, such as the Frequent Pattern Growth (FP-Growth) algorithm, can identify operation sequences (operation trajectories) with high frequency and strong correlation. Assume a user's operation sequence is... The processor can calculate each operation Frequency of occurrence and the support for sequence patterns To determine the high-frequency operating mode.

[0183]

[0184]

[0185] in, Indicates operation The number of times it appears in all operation records. The number of operation sequences for all users.

[0186] S1113, Agent Generation and Recommendation.

[0187] Based on the identified high-frequency operation patterns, the system automatically generates corresponding intelligent agent editing suggestions. Users can then operate based on these suggestions, quickly creating intelligent agents that meet their needs. For example, for a frequently performed three-step operation... The system will automatically generate an agent template that meets the user's personalized needs. Users can create their own agents through simple editing, which greatly reduces the complexity of manual configuration.

[0188] The second method, similar user agent recommendation, includes steps S1121-S1123.

[0189] S1121, Feature Extraction. In the similar user agent recommendation module, the system first analyzes users' personalized preferences to extract user features from the log data. These preferences include users' behavioral habits, interests, frequently used applications and functions, etc. To ensure privacy and security, the system anonymizes this data and abstracts it into feature vectors. Each feature This corresponds to a user's preference dimension, such as usage frequency or preferred type. (And the anonymization process follows.)

[0190] S1122, Similarity Calculation. The system uses feature vectors to calculate the similarity of personalized preferences between users. The similarity calculation method in this application embodiment may include cosine similarity and Euclidean distance. Taking cosine similarity as an example, similarity... It can be represented as:

[0191]

[0192] in, and users respectively and The feature vector. The closer this similarity value is to 1 (the preset value), the more similar the personalized preferences of the users are.

[0193] S1123, Agent Recommendation.

[0194] After calculating the similarity between users, the system uses clustering algorithms, such as K-means clustering (K-means), to group similar users and recommends corresponding agents based on their operational data and usage habits. For example, if a group of users frequently uses a specific function in an application and generates a specific agent template, the system will recommend these agents to the users, allowing them to quickly obtain personalized agents and thus improving the practicality and usability of the agents.

[0195] The recommendation algorithm of this application provides an agent suitable for personalized service recommendation scenarios. The system can automatically recommend suitable agents based on the user's operating habits and historical data. For example, for users who enjoy sports, the system can recommend automated sports-related tasks, such as automatically recording exercise data and sending reminders to drink water after exercise. This readily meets the user's personalized needs for agents.

[0196] It should be noted that throughout the entire recommendation process of the intelligent agent described above, the system consistently adheres to strict data privacy protection principles. All collected user data undergoes anonymization to ensure the removal of sensitive user identity information. Furthermore, encrypted transmission and storage mechanisms are employed during recommendation execution to prevent data leakage or unauthorized access. Additionally, users can choose not to have their information used; for example, when retrieving user log data, the mobile interface prompts the user about which information is being retrieved and its purpose. Users can also choose not to participate in data feedback and analysis, further enhancing their control over their personal data.

[0197] Therefore, through the collaborative work of the two recommendation modules, the trajectory recommendation strategy based on the co-occurrence mechanism can effectively solve the problems of insufficient personalization and low recommendation accuracy in traditional intelligent agent recommendation systems. The high-frequency operation intelligent agent recommendation module provides users with intelligent intelligent agent suggestions based on their own operating habits. The similar user intelligent agent recommendation module further improves the accuracy and practicality of recommendations by mining the commonalities among users. The recommendation method of this application embodiment not only optimizes the user experience but also ensures that user privacy and data security are fully protected throughout the entire process.

[0198] In the above embodiments, the construction process of the operation trajectory (smart agent) has been described in detail. In the following embodiments, the interaction method of the embodiments of this application will be described in detail with reference to the accompanying drawings.

[0199] Once the intelligent agents constructed using the above method are established, the mobile phone can store these intelligent agents, or they can be stored in the cloud. When stored in the cloud, the user can operate the mobile phone to input corresponding commands. The mobile phone will then download the corresponding intelligent agent from the cloud based on the user's commands and execute the intelligent agent to automatically complete the user's desired task.

[0200] In some embodiments, the process of constructing the operation trajectory and the device that actually executes the operation trajectory may be different electronic devices. The device that executes the operation trajectory is an electronic device that interacts directly with the user. For example, the electronic device may be, but is not limited to, a mobile phone, a smart TV, a wearable device, a tablet computer, a computer with wireless transceiver capabilities, a virtual reality (VR) device, an augmented reality (AR) device, a wireless device in industrial control, a wireless device in self-driving cars, etc.

[0201] The following embodiments use a mobile phone as an example for illustration.

[0202] refer to Figure 12 , Figure 12 A flowchart illustrating the interaction method according to an embodiment of this application is shown. This method can be executed by a mobile phone. Figure 12 As shown, this interaction method includes, but is not limited to, the following steps:

[0203] S1210, Receive user instructions (first operation).

[0204] In the embodiments of this application, user instructions are used to instruct electronic devices to perform target tasks. These user instructions can be voice instructions, touchscreen instructions, or text instructions, etc. The target task can be as described in the above application scenarios, such as automatically turning off airplane mode every morning, playing music, automatically processing emails, scheduling meetings, etc., and will not be elaborated further here.

[0205] In some embodiments, after receiving a user's voice command, the mobile phone can display the corresponding text content in the dialog box between the user and the voice assistant, or it can choose not to display it and simply reply to the user via voice. Taking the current display interface when the mobile phone receives a user's voice command as the main interface as an example, after receiving the user's voice command, the main interface will display a dialog box between the user and the voice assistant, and the dialog box will contain the content of the voice command.

[0206] In some embodiments, the first operation may be any of the following operations.

[0207] The first action can be a pre-defined action (a specific action), such as adding items to a shopping cart in an application (selecting an item), or opening the application, such as a map application. Alternatively, the user might take their phone to a designated location. Based on these actions, the electronic device can determine the user's intent. For example, when a user selects an item, it can infer that the user wants to buy it and recommend a payment agent, or directly display the payment interface. When a user opens a map application, it can infer that the user wants navigation and recommend possible addresses. When the user reaches a designated location, it can infer that the user might need a taxi and recommend a taxi agent, or directly select a vehicle and ask the user for confirmation, etc.

[0208] The first operation could be a selection of an agent within a specific application. This application could provide task-related agents, representing entities capable of executing actions to complete a target task. For example, in an agent store application, a user opens the application and selects an agent tag for sharing contacts. These agent tags could be displayed in recommended locations within the application's page based on the user's historical action data.

[0209] The first action could be entering keywords into the search area. For example, a user can search for a contact to share using the search function. At this point, the system can confirm that the user wants to share the contact and display recommended smart agent tags on the interface, or directly open the corresponding application and wait for sharing.

[0210] The first operation could be receiving an agent shared by another user device. For example, agent sharing between two devices.

[0211] The first operation can also be in the scenario of multi-device collaborative requests, where it receives operations from intelligent agents shared by other devices.

[0212] Through the above operations, intelligent agents can be provided to users in various scenarios, providing convenience and reducing the time wasted by user editing.

[0213] S1220, in response to the first operation, recommend an intelligent agent (first intelligent agent) to the first user, the intelligent agent being used to perform the first task.

[0214] After determining the user's target task (first task), the electronic device determines the target operation trajectory corresponding to the target task from the pre-stored operation trajectory.

[0215] The process of constructing the operation trajectory has been described in detail in the above embodiments. For details, please refer to the operation trajectory construction method in the above embodiments. It will not be repeated here.

[0216] In some embodiments, recommending an agent to a first user may include the following scenarios:

[0217] When a user opens the Smart Agent Store, multiple smart agents are recommended to the user. Among these smart agents, the user's desired smart agent (the first smart agent) can be included.

[0218] In some scenarios, electronic devices recognize a user's intent based on their first action. For example, when a user adds an item to their shopping cart in a store, the electronic device recognizes that the user wants to purchase the item and, based on that intent, recommends an agent to open the payment page. In some embodiments, the first action is used to receive a message from a second user and present the user with an agent that the second user shared with the first user via the message.

[0219] The first task can be a multi-user collaborative task. This task requests the first user and the second user to collaborate on the first task based on the first intelligent agent. This enables the sharing of intelligent agents under multi-task collaboration.

[0220] In the embodiments of this application, the mobile phone can set a title or label for each target operation trajectory. Upon receiving a user instruction, the user instruction can be analyzed and compared with the label or title of the operation trajectory to determine the intelligent agent that matches the instruction.

[0221] S1230, receive the second operation.

[0222] In some embodiments, the second operation may be a viewing operation, a selection operation, a confirmation operation, or a modification operation on the intelligent agent.

[0223] S1240, in response to the second operation, presents the operation trajectory of the agent performing the first task.

[0224] This process can be executed by the phone after the user confirms the action. Alternatively, the user can set the default to no confirmation in the settings, so that the phone directly executes the found smart agent after receiving the instruction.

[0225] In some embodiments, presenting the operation trajectory of an intelligent agent performing a first task may include: displaying the operation trajectory through an interface (first interface); the interface may also include at least one control for modifying the operation trajectory. Modifying the operation trajectory may include any one of manually editing the operation trajectory, automatically optimizing the operation trajectory, or re-recording the operation trajectory.

[0226] In the embodiments of this application, the operation trajectory is displayed in text form according to the execution order. This display is more in line with the user's operating habits and is easier for the user to edit.

[0227] According to the interaction method of the embodiments of this application, since the intelligent agent can be executed across applications and has high execution accuracy and a high success rate, the user experience is greatly improved.

[0228] The interaction method of this application embodiment will be further described below with reference to the operation of the mobile phone interface.

[0229] refer to Figure 13 , Figure 13 A schematic diagram illustrating the user interface interaction between the user and the mobile phone is provided, representing an embodiment of this application. For example... Figure 13 As shown, users can directly input commands via voice (first command). For example, a user can input the target task via voice: "Share contact information for Li Xiang to my WeChat." TM After receiving a voice command, the mobile phone analyzes the command and identifies the corresponding agent. The dialog box 1311 on the main interface 1310 displays the text "The following agents have been recommended for you," and provides user options such as "Execute" and "View." When the user clicks "View," interface 1320 is displayed, showing the agent's steps sequentially, for example, executing steps 1-6, each with its corresponding execution statement. The user can check the accuracy of these steps and, if needed, can click the "Edit" function (control) on interface 1320 to edit each step. The user can also click the "Auto Optimize" (control) on interface 1320 to optimize the agent; the optimization process can be referenced above. Figure 10The optimization process can be optimized. Alternatively, users can click the "Re-record" function (control) in interface 1320 to manually complete the target task. The phone will record the user's actions, and the recorded video can be used to obtain an intelligent agent based on a large language model, which will then be displayed again in the execution steps. This allows for a faster and more accurate acquisition of the desired intelligent agent. Furthermore, users can directly click the "Try it" function in interface 1320, and the phone will respond to the command, starting to execute each step according to the intelligent agent's description to complete the user's target task. In other embodiments, users can also manually input commands, which can be text input or input through clicks, swipes, etc. For example, a user can open the intelligent agent store and select a suitable intelligent agent based on the store's recommendations.

[0230] refer to Figure 14 , Figure 14 A schematic diagram illustrating a user interface for interaction with a mobile phone, representing another embodiment of this application, is shown. Figure 14 As shown, the user opens the Agent Store (second instruction), displaying interface 1410. In interface 1410, the phone recommends agents to the user based on their personalized needs, specifically the recommendation algorithm of the sharing mechanism mentioned above. For example, under the "You May Like" module, there are options like "Post to Moments" and "Share Contacts to WeChat". TM The corresponding task tags are labeled "...". These task tags correspond to intelligent agents that can complete the corresponding target tasks. When the user clicks (the second instruction) "Share contact to WeChat..." TM The phone receives the click and displays interface 1420. Interface 1420 displays the same content as interface 1320, showing the various steps of the agent in sequence for easy viewing and editing by the user. When the user clicks "Try," the phone executes the aforementioned steps, completing each step according to the description in the agent until the target task in the instruction is completed. The user can also click "Add" to add steps, etc.

[0231] In some embodiments, such as Figure 14 The main interface 1410 of the Agent Store shows that when the desired agent is not available in the store, the user can add a new agent by clicking the "+" widget 1411. For example, based on the user's click, the phone can recommend more agents to provide the user with more choices.

[0232] The following section, with reference to the interface diagram, explains the steps and corresponding interface for executing a smart agent on a mobile phone.

[0233] refer to Figure 15 , Figure 15A schematic diagram illustrating the interface of a mobile phone executing an intelligent agent, according to an embodiment of this application, is shown. Figure 15 As shown, the phone received the user's voice command: "Share Li Xiang's information to my WeChat". TM For example, the mobile phone receives the user's voice command (the first command), as described above. Figure 13 and Figure 14 After the user clicks "Try," the phone automatically opens the Contacts app on the home screen (1510), enters the Contacts interface (1520), selects "Li Xiang," and accesses the details interface (1530). Selecting "More Options," finding "Share Contacts" on interface (1540), leads to interface (1550). In interface (1550), selecting "WeChat"... TM (Third-party application), and in the newly displayed interface 1560, select "Send to friend" to enter WeChat. TM In the contacts interface (1570), select the contact's name "Li Xiang," and then select "Send" in interface (1580). This will take you to the chat interface between Li Xiang and me (1590). At this point, the message from Li Xiang has been successfully shared to my WeChat. TM This command is completed automatically by the mobile phone, saving users time and effort and greatly improving the user experience.

[0234] This application also provides an electronic device, including:

[0235] Memory, used to store instructions executed by one or more processors of the device, and

[0236] Processor, used to execute the above embodiments combined with Figures 2 to 15 The method explained.

[0237] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the above-described embodiments. Figures 2 to 15 The method explained.

[0238] This application also provides a computer program product containing instructions that, when run on an electronic device, cause a processor to execute the above-described embodiments. Figures 2 to 15 The method explained.

[0239] Now for reference Figure 16 The diagram shown is a block diagram of a SoC (System on Chip) 1600 according to an embodiment of this application. Figure 16 In the diagram, similar components share the same reference numerals. Additionally, dashed boxes are an optional feature for more advanced SoCs. Figure 16In this embodiment, SoC 1600 includes: an interconnect unit 1650 coupled to an application processor 1610; a system agent unit 1680; a bus controller unit 1690; an integrated memory controller unit 1640; one or more coprocessors 1620, which may include integrated graphics logic, an image processor, an audio processor, and a video processor; a static random access memory (SRAM) unit 1630; and a direct memory access (DMA) unit 1660. In one embodiment, the coprocessor 1620 includes a dedicated processor, such as, for example, a network or communication processor, a compression engine, a general-purpose computing on GPU (GPGPU), a high-throughput many integrated core (MIC) processor, or an embedded processor.

[0240] The static random access memory (SRAM) cell 1630 may include one or more computer-readable media for storing data and / or instructions. The computer-readable storage medium may store instructions, specifically, temporary and permanent copies of those instructions. These instructions may include, when executed by at least one unit in the processor, causing the SoC 1600 to perform the methods explained in the above embodiments, as detailed in the above embodiments. Figure 2 and Figure 15 The methods explained will not be repeated here.

[0241] Various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or combinations of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0242] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0243] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0244] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other propagation signals. Therefore, machine-readable media include any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0245] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0246] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0247] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0248] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made thereto without departing from the spirit and scope of this application.

Claims

1. An interaction method applied to an electronic device, characterized in that, include: Receive a first operation from a first user on the electronic device; In response to the first operation, a first intelligent agent is recommended to the first user, and the first intelligent agent is used to perform the first task; Receive a second operation from the first user on the electronic device; In response to the second operation, the operation trajectory of the first agent executing the first task is displayed on the first interface; The operation trajectory includes multiple modifiable steps presented in natural language according to the execution order.

2. The method according to claim 1, characterized in that, The presentation of the operation trajectory of the first intelligent agent executing the first task includes: The operation trajectory is displayed on the first interface; The first interface also includes at least one control for modifying the operation trajectory.

3. The method according to claim 2, characterized in that, The modification of the operation trajectory includes at least one of the following: The operation trajectory can be manually edited; The operation trajectory is automatically optimized; as well as The operation trajectory was re-recorded.

4. The method according to any one of claims 1-3, characterized in that, The second operation is a viewing operation, selection operation, confirmation operation, or modification operation of the first intelligent agent.

5. The method according to any one of claims 1-4, characterized in that, The step of recommending the first intelligent agent to the first user includes: In the agent store, multiple agents are recommended to the first user, including the first agent.

6. The method according to any one of claims 1-5, characterized in that, The step of recommending a first intelligent agent to the first user in response to the first operation includes: Based on the first operation, identify the intent of the first user; Based on the stated intent, the first intelligent agent is recommended to the first user.

7. The method according to any one of claims 1-5, characterized in that, The first operation is used to instruct the electronic device to perform the first task.

8. The method according to any one of claims 1-5, characterized in that, The first operation is to enter a search term in the search box.

9. The method according to any one of claims 1-5, characterized in that, The first operation is dialogue input in the smart assistant.

10. The method according to any one of claims 1-5, characterized in that, The first operation is used to receive a message from a second user, and the step of recommending the first agent to the user includes: presenting the first agent shared by the second user to the first user through the message to the user.

11. The method according to claim 11, characterized in that, The first task is a multi-user collaborative task, and the message is used to request the first user to collaborate with the second user to execute the first task based on the first intelligent agent.

12. A method for constructing an operation trajectory, characterized in that, include: Obtain the dataset corresponding to the application. The dataset includes a list of controls corresponding to multiple pages in the application and a specification of the application. The list of controls is used to record the executable controls in the pages, and the specification at least describes the operation corresponding to the control. Based on the dataset, a user interface transition map is determined; By inputting the user interface transfer map and the instruction manual into the large model, an operation trajectory for completing the target task is obtained.

13. The method according to claim 12, characterized in that, Determining the user interface transition map based on the dataset includes: Obtain all pages of the application and a list of controls corresponding to each page, and use the content and control list corresponding to each page as nodes of the user interface transition graph; When transitioning between pages, obtain the corresponding control information and operation type, and use the control information and operation type as edges of the user interface transition graph to obtain the user interface transition graph.

14. The method according to claim 13, characterized in that, The step of obtaining all pages of the application and the list of controls corresponding to each page includes: Launch the application and retrieve the list of controls corresponding to the current page; Operate each control one by one according to the instructions described in the manual. The content corresponding to the new page that appears after each operation is saved, and the control list corresponding to the new page is obtained. The controls in the control list corresponding to the new page are operated one by one until the control lists corresponding to all applications of the application are traversed, so as to obtain all pages of the application and the control list corresponding to each page.

15. The method according to any one of claims 12-14, characterized in that, Also includes: A simulation verification environment is constructed to verify all the operation trajectories. If the verified operation trajectory meets the expected execution requirements, the operation trajectory that meets the expected value is retained.

16. The method according to claim 15, characterized in that, The verification of all the aforementioned operation trajectories includes: In a simulated environment, based on the description of the operation trajectory, the actual operation scenario is simulated, and the steps corresponding to the operation trajectory are executed step by step. If each step of the current operation trajectory can be executed according to the preset requirements, then the operation trajectory is determined to meet the expected execution requirements.

17. The method according to any one of claims 12-14, characterized in that, The operation trajectory that meets the expected execution requirements is input into the large model, and the operation trajectory is optimized.

18. The method according to claim 17, characterized in that, The optimization of the operation trajectory includes: Acquire historical operation behavior data for completing the target task and result data for executing the operation trajectory, wherein the result data includes the state generated during the execution of the operation trajectory and the action data during the execution of the operation trajectory; The user's historical operation behavior data and the result data are input into the trajectory optimization model to obtain the optimized operation trajectory.

19. The method according to claim 17, characterized in that, The method further includes: evaluating the optimized operation trajectory. When the operation trajectory completes the target task or completes the preset intermediate steps, the optimized result is labeled with a reward tag. When the operation trajectory encounters an error or fails to achieve the expected target task, the optimized result is labeled with a penalty tag.

20. The method according to claim 18, characterized in that, The states include: the content displayed on the current page corresponding to the application, the text entered by the user, or the activation state of the control.

21. The method according to claim 18, characterized in that, The action data includes: clicking, swiping, or entering text.

22. The method according to claim 18, characterized in that, The historical operation behavior data includes: the user's historical operation behavior and the historical operation behavior of other users, wherein the similarity between the historical operation behavior of other users and the historical operation behavior of the user meets a preset value.

23. The method according to claim 12, characterized in that, The application's specification includes at least one of the following: a summary of the user interface, a description of the functions of the controls, a page navigation diagram, and a task operation sequence.

24. An electronic device, comprising: A memory for storing instructions executed by one or more processors of an electronic device, and a processor for performing the method of any one of claims 1 to 11 or the method of any one of claims 12 to 23.

25. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the method of any one of claims 1 to 11 or the method of any one of claims 12 to 23.

26. A computer program product containing instructions, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the method of any one of claims 1 to 11 or the method of any one of claims 12 to 23.

Images