Interaction method and apparatus, electronic device, and storage medium

Abstract

This application proposes an interaction method, device, electronic device, and storage medium, relating to fields such as artificial intelligence and smart cockpits. The method includes: in response to receiving a task instruction, invoking an intelligent agent to execute an interactive task matching the task instruction based on the task instruction and the screen content of the user interface; during the execution of the interactive task, in response to receiving a first target event that conflicts with the actual state of the intelligent agent, determining a first target state matching the first target event; and updating the state of the intelligent agent to the first target state. This achieves closed-loop adaptive adjustment of task execution, effectively avoiding misoperations or task failures caused by state mismatch, thereby significantly improving the robustness, security, task success rate, and user experience of the intelligent agent in real human-machine coexistence scenarios.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and smart cockpit, and in particular to an interaction method, device, electronic device and storage medium. Background Technology

[0002] With the rapid development of Large Language Model (LLM) and Multimodal Large Language Model (MLLM) technologies, agents based on large models have made significant progress in task planning, task orchestration, and external tool invocation. However, in real-world interactive scenarios oriented towards Graphical User Interfaces (GUIs), once an agent starts a task, it executes according to a pre-defined process, unable to dynamically adapt to environmental changes. This severely restricts the robustness, security, and user experience of agents in open, dynamic, and human-machine coexisting real-world scenarios. Summary of the Invention

[0003] This application proposes an interaction method, apparatus, electronic device, and storage medium to at least partially solve one of the technical problems in the related art.

[0004] One embodiment of this application proposes an interaction method, including:

[0005] In response to receiving a task instruction, the agent is invoked to execute an interactive task matching the task instruction, based on the task instruction and the screen content of the user interface. During the execution of the interactive task, in response to receiving a first target event that conflicts with the actual state of the agent, a first target state that matches the first target event is determined; Update the state of the agent to the first target state.

[0006] Another embodiment of this application proposes an interactive device, including: An execution module is used to respond to a received task instruction by invoking an agent to perform an interactive task matching the task instruction based on the task instruction and the screen content of the user interface. The determination module is used to determine a first target state that matches the first target event in response to receiving a first target event that conflicts with the actual state of the agent during the execution of the interactive task. The update module is used to update the state of the agent to the first target state.

[0007] In another aspect of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the interaction method as described in the foregoing aspect.

[0008] Another embodiment of this application proposes a non-transitory computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the interaction method as described in the foregoing aspect.

[0009] Another aspect of this application provides a computer program product having a computer program stored thereon, which, when executed by a processor, implements the interaction method as described in the foregoing aspect.

[0010] The interaction method, device, electronic device, and storage medium proposed in this application introduce a conflict management mechanism for the agent's state. During the execution of the interaction task, the agent listens for or receives events in real time. When a first target event that conflicts with the agent's current actual state is received, the agent's state is dynamically updated to a first target state that matches the first target event. This achieves closed-loop adaptive adjustment of task execution, effectively avoiding misoperation or task failure caused by state mismatch. As a result, the robustness, security, task success rate, and user experience of the agent in real human-machine coexistence scenarios are significantly improved.

[0011] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0012] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart illustrating an interactive method provided for an exemplary embodiment of this application; Figure 2 A flowchart illustrating another interaction method provided for an exemplary embodiment of this application; Figure 3 A flowchart illustrating yet another interaction method provided for an exemplary embodiment of this application; Figure 4 A flowchart illustrating another interaction method provided for an exemplary embodiment of this application; Figure 5 A flowchart illustrating another interaction method provided for an exemplary embodiment of this application; Figure 6 An architectural illustration of an interactive system provided for an exemplary embodiment of this application. Figure 1 ; Figure 7 A schematic diagram illustrating a state management principle provided for an exemplary embodiment of this application; Figure 8 A schematic diagram illustrating the relationship between system event operations, operation sources, and recovery strategies, provided for an exemplary embodiment of this application; Figure 9 A schematic diagram illustrating a conflict management method provided for an exemplary embodiment of this application; Figure 10 A schematic diagram illustrating a state change principle provided for an exemplary embodiment of this application; Figure 11 An architectural illustration of an interactive system provided for an exemplary embodiment of this application. Figure 2 ; Figure 12 A schematic diagram of the structure of an interactive device provided for an exemplary embodiment of this application; Figure 13 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this application; Figure 14 This is a block diagram illustrating a vehicle according to an exemplary embodiment. Detailed Implementation

[0013] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0014] It should be noted that the acquisition, storage, use, and processing of data in this application comply with relevant laws and regulations and do not violate public order and good morals.

[0015] It should also be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0016] The interaction method, apparatus, electronic device, and storage medium of this application are described below with reference to the accompanying drawings. Before specifically describing the embodiments of this application, for ease of understanding, commonly used technical terms in this application will first be introduced: A UI agent is an intelligent agent that interacts within a user interface (UI) environment. On electronic devices with displays (such as mobile phones, smart cockpits, televisions, and personal computers), it semantically understands the user's interaction needs and automatically interacts with the user interface based on screen understanding to complete user tasks. This not only improves the user experience but also avoids accidental user interface operations, thus increasing interaction efficiency. An intelligent agent must possess at least the following core capabilities: 1. User interface awareness: Possesses user interface design paradigms and interaction conventions, and is able to identify and understand the visual features and functional semantics of common interface elements such as buttons, input boxes, lists, and labels; 2. Positioning capability: Possesses pixel-level or sub-pixel-level interface element positioning capability to accurately complete fine-grained user interface operations such as clicking, swiping, long pressing, and text input; 3. Planning and reasoning ability: Combining task description information in natural language form with the current user interface state, it can realize intent parsing, multi-step task planning and cross-modal reasoning, has good scenario generalization ability, and can pay attention to task context and operation details during execution; 4. Decision-making and reflection capabilities: Possessing adaptive decision-making and reflection capabilities in dynamic or unexpected environments, able to identify and handle abnormal situations such as pop-up ads, permission prompts, network latency, and operation failures, and adjust subsequent behavioral strategies through feedback mechanisms to ensure the robustness and completion rate of tasks.

[0017] Figure 1 This is a flowchart illustrating an interactive method provided for an exemplary embodiment of this application.

[0018] It should be noted that the interaction method of this application embodiment can be applied to an interactive device. In some possible embodiments, the interactive device can be configured in an electronic device so that the electronic device can perform interactive functions. Additionally, in some possible embodiments, the interactive device can also be software within the electronic device.

[0019] The electronic devices mentioned include, but are not limited to, terminals, vehicles, and personal computers. A terminal is a user-side entity used to receive or transmit signals, such as a mobile phone. Terminals can also be called terminal equipment (UE), user equipment (UE), mobile station (MS), mobile terminal equipment (MT), etc. Terminals can be communication-enabled vehicles, smart cars, mobile phones, wearable devices, tablets, computers with wireless transceiver capabilities, virtual reality (VR) terminals, augmented reality (AR) terminals, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, etc. The embodiments of this application do not limit the specific technology or device form used in the terminal.

[0020] The vehicle can be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or other types of vehicles, and this application embodiment does not limit this.

[0021] like Figure 1 As shown, the interaction method may include the following steps S101 to S103: Step S101: In response to receiving a task instruction, the agent is invoked to execute an interactive task matching the task instruction based on the task instruction and the screen content of the user interface.

[0022] Task instructions can be natural language descriptions of tasks input by users for interactive tasks. Input methods for task instructions include, but are not limited to: touch input (such as swiping, clicking, etc.), keyboard input, voice input, etc.

[0023] The intelligent agent can be deployed on the electronic device side (or client), or it can be deployed in the cloud (or server). This application embodiment does not limit this.

[0024] The screen content includes, but is not limited to: screenshots of the user interface, positional information and semantic information of interface elements on the screen, etc. The positional and semantic information of interface elements on the screen can be obtained through visual recognition of the screenshots, or through parsing the underlying structural data associated with the user interface; this application does not impose any limitations on these methods.

[0025] The underlying structure data includes, but is not limited to, at least one of the following: HyperText Markup Language (HTML) files, Extensible Markup Language (XML) layout files, Accessibility Tree, or JavaScript Object Notation (JSON) configuration files.

[0026] Interactive tasks are those used to interact with the user interface. For example, interactive tasks include, but are not limited to: resource search tasks (such as text search tasks, video search tasks, and audio search tasks), user-generated content (UGC) publishing tasks, navigation tasks, ticket purchase tasks, and food ordering tasks. It should be noted that this application does not specifically limit the field to which the interactive tasks belong.

[0027] In this embodiment of the application, upon receiving a task instruction input by a user, an intelligent agent can be invoked to execute an interactive task matching the task instruction based on the task instruction and the screen content of the user interface.

[0028] Step S102: During the execution of the interactive task, in response to receiving a first target event that conflicts with the actual state of the agent, a first target state that matches the first target event is determined.

[0029] The actual state of the agent includes, but is not limited to, the following states: initial state, execution state (active state), paused state (temporary waiting state), stopped state, and completed state (end state).

[0030] The first target state includes, but is not limited to: a paused state used to instruct the agent to suspend execution, and a stopped state used to instruct the agent to stop execution.

[0031] In the embodiments of this application, during the process of the intelligent agent performing interactive tasks, event reception (or event listening) can be performed. If a first target event that conflicts with the actual state of the intelligent agent is detected or received, a first target state that matches the first target event can be determined.

[0032] As one possible implementation, the matching relationship between different states and event types can be pre-configured. Thus, in this application, the matching relationship can be queried based on the event type to which the first target event belongs, in order to determine the first target state that matches the event type to which the first target event belongs.

[0033] Step S103: Update the state of the agent to the first target state.

[0034] As an example, when the agent is deployed on an electronic device, the electronic device can directly update the agent's state to the first target state.

[0035] As another example, when the agent is deployed on the cloud side, the electronic device can report the first target event or the first target state to the cloud so that the cloud can update the state of the agent to the first target state.

[0036] The interaction method in this application introduces a conflict management mechanism for the agent's state. During the execution of the interaction task, it performs real-time monitoring or reception of events. When a first target event that conflicts with the agent's current actual state is received, the agent's state is dynamically updated to a first target state that matches the first target event. This achieves closed-loop adaptive adjustment of task execution, effectively avoiding misoperation or task failure caused by state mismatch. As a result, it significantly improves the robustness, security, task success rate, and user experience of the agent in real human-machine coexistence scenarios.

[0037] As one possible implementation method, Figure 2 A flowchart illustrating another interaction method provided for an exemplary embodiment of this application.

[0038] It should be noted that this interaction method can be executed alone, or it can be executed together with any embodiment or possible implementation in the embodiment of this application, or it can be executed together with any technical solution in the related technology. The embodiments of this application do not limit this.

[0039] like Figure 2 As shown, the interaction method may include the following steps S201 to S202: Step S201: In response to receiving a task instruction, the agent is invoked to execute an interactive task matching the task instruction based on the task instruction and the screen content of the user interface; wherein, the user interface includes an operation area and a status area.

[0040] It should be noted that the explanation of step S201 can be found in the relevant description in any embodiment of this application, and will not be repeated here.

[0041] In any embodiment of this application, the user interface includes an operation area and a status area, wherein the status area is different from the operation area, that is, the user interface includes a status area and an operation area that is different from the status area.

[0042] The operation area refers to the region in the user interface used to display the specific operations and processes performed by the agent on the target application, allowing users to intuitively understand "what the agent is doing" and improving the observability and comprehensibility of automated operations. The status area refers to the region in the user interface used to present the agent's current operating status, decision context, and other prompts.

[0043] In any embodiment of this application, the operation area and / or status area may be selectively rendered to the user interface, or they may be generated only internally within the system and not presented to the user. This application does not limit this.

[0044] For example, both the operation area and the status area can be rendered onto the front-end user interface so that users can intuitively view the visual content presented in the operation area and the status area.

[0045] For example, only the operation area can be rendered to the foreground user interface, while the status area is maintained internally by the system (invisible to the user).

[0046] For example, only the status area can be rendered to the foreground user interface, while the operation area is generated internally by the system but not displayed.

[0047] For example, the operation area and status area may not be rendered to the foreground user interface, such as when running in headless mode or background automation mode. Headless mode refers to a mode in which the program runs without a graphical user interface.

[0048] Step S202: During the execution of the interactive task, at least one of the following is simultaneously displayed in the user interface: In the operation area, operation information of the agent on the target application associated with the interactive task is displayed; In the operation area, interactive animation is displayed, wherein the interactive animation is associated with the operation performed by the agent on the target application; In the status area, prompt information matching the status of the agent is displayed.

[0049] The operational information includes, but is not limited to, the application page and page elements used by the intelligent agent.

[0050] Interactive animations are dynamic visual feedback triggered by the actions of the intelligent agent, including but not limited to: click animations, light animations, hover animations, drag feedback animations, etc.

[0051] The states that an agent is in include, but are not limited to, the following states: initial state, execution state (active state), paused state (temporary waiting state), stopped state, and completed state (end state).

[0052] The prompts are related to the state of the agent, and different states correspond to different prompts. The display formats of these prompts include, but are not limited to, text, images, and animations.

[0053] In this embodiment of the application, during the process of the intelligent agent performing an interactive task, at least one of the following can be displayed synchronously in the user interface: The first item is to display the agent's operation information on the target application associated with the interactive task in the operation area of ​​the user interface.

[0054] The second aspect involves displaying interactive animations in the user interface's operation area. These animations are associated with the actions performed by the agent on the target application. Different actions can have different associated animations; for example, the interactive animations associated with a click action are different from those associated with a search action.

[0055] Interactive animations can be displayed in the areas operated by the agent.

[0056] For example, the association between different operation types and interactive animations can be preset. Thus, in this application, the association can be queried according to the operation type performed by the agent on the target application to determine the matching interactive animation and display the interactive animation in the operation area.

[0057] The third item is to display prompts in the status area of ​​the user interface that match the state of the agent.

[0058] As an application scenario, users can input task commands using voice interaction when using electronic devices such as vehicles and mobile phones. For example, if the task command is "Help me search for song A in application (APP) 1", then the intelligent agent can be invoked to automatically control APP 1 to search for song A in APP 1. As another example, if the task command is "Help me post picture B in APP 2 (social APP)", then the intelligent agent can be invoked to automatically control APP 2 to post picture B in APP 2.

[0059] In any embodiment of this application, the display method of the prompt information is, for example, to determine the content indicated by the prompt information according to the display constraints corresponding to the status area, and to display the content indicated by the prompt information in the status area.

[0060] The display constraints include, but are not limited to, at least one of the following: the spatial size parameters of the status area in the user interface, the device's movement status, and the historical interaction behavior associated with the status area.

[0061] Among them, spatial dimension parameters include, but are not limited to: area size, area ratio, height, width, etc.; device movement status includes, but is not limited to: movement speed, movement acceleration, etc.; historical interaction behavior includes, but is not limited to: click behavior, pause behavior, swipe operation, long press operation, etc.

[0062] The presentation formats of the prompts include, but are not limited to, static presentation formats and dynamic presentation formats.

[0063] As an example, when the spatial size parameter is relatively large, and / or the device movement speed is relatively low, and / or when the user frequently interacts with the state area based on historical interaction behavior, the content indicated by the prompt message should include both the agent's current state and the subtask description information at the current time step in the execution path associated with the interaction task. For example, if the visible space of the state area is insufficient to fully statically display the prompt message, dynamic display methods such as scrolling or page turning can be used to display the prompt message.

[0064] The execution path is generated by the agent based on task instructions and the screen content of the user interface, and includes subtask description information at at least one time step. For example, taking the interactive task of buying movie tickets as an example, the execution path may include: Open APP3 (a ticketing app) → Select "Movies" (tab) → Enter movie name → Select to buy tickets → Select cinema → Select date and showtime → Select seat and pay. Taking the current time step as the 3rd time step as an example, the prompt information includes: the agent's current state "execution state" and the subtask description information "Enter movie name" at the current time step.

[0065] For example, taking a vehicle as an example of an electronic device, when the spatial dimensions of the status area (such as area size and area ratio) are relatively large, ensuring that the prompt information includes both the agent's current state and the sub-task description at the current time step allows full utilization of the larger display area. This clearly presents the agent's task execution progress (e.g., "Enter movie name") to the driver without obstructing their view, enhancing task interpretability, reducing user confusion about system behavior, and improving human-machine collaboration efficiency. Conversely, when the device's movement speed (i.e., the vehicle's speed) is relatively low, the driver's attention resources are relatively abundant. The timely confirmation of the prompt message includes both the agent's current state and the sub-task description at the current time step. This helps drivers to promptly confirm or intervene in the agent's operations (such as modifying ticket bookings), improving interaction safety and task completion reliability. When a user's high-frequency interaction state area is identified based on historical interaction behavior, it indicates that the user has a high level of attention to or willingness to operate on that state area. In this case, ensuring that the prompt message includes both the agent's current state and the sub-task description at the current time step can reduce the user's cognitive load, avoid repeated queries, achieve personalized information presentation on demand, and improve user experience consistency.

[0066] As another example, when the spatial size parameter is relatively small, and / or the movement speed is relatively large, and / or the user's low-frequency interaction with the state zone is determined based on historical interaction behavior, the content indicated by the prompt message includes: the state of the agent. For example, when the prompt message is presented in a static format, the content indicated by the prompt message may only include the state of the agent; while when the prompt message is presented in a dynamic format (such as scrolling, page turning, etc.), the content indicated by the prompt message may also include both the state of the agent and the subtask description information at the current time step in the execution path associated with the interaction task.

[0067] For example, taking a vehicle as an example of an electronic device, when the spatial dimensions of the status area (such as area size and area ratio) are relatively small, ensuring that the content indicated by the prompt message includes the status of the intelligent agent can avoid visual interference caused by information overload, ensure that key status information is prominently visible, maintain the driver's focus, and improve driving safety. When the device's moving speed (i.e., the vehicle's driving speed) is relatively low, the driver needs to concentrate on the road environment. In this case, limiting the content indicated by the prompt message to include the status of the intelligent agent can significantly reduce the driver's cognitive burden on non-driving tasks, reduce the time of gaze deviation, and effectively ensure the safety of vehicle driving. When the user's low-frequency interaction status area is determined based on historical interaction behavior, it indicates that the user's current attention to this status area is low. In this case, adopting a minimal information presentation strategy and ensuring that the content indicated by the prompt message includes the status of the intelligent agent can both meet the user's basic status perception needs and avoid irrelevant details from interfering with the main driving task, achieving efficient use of interface resources and seamless interaction.

[0068] In summary, this application achieves adaptive granular control of prompt information content by introducing multi-dimensional display constraints (including spatial dimensions, device movement status, and user interaction activity) and combining them with a threshold mechanism. This dynamically balances information integrity and interface simplicity, avoids cognitive interference caused by information overload, and provides high-information-density feedback under safe driving conditions (such as low speed and stationary conditions). It also improves the transparency of the agent's state and the interpretability of the task, providing a reliable information presentation foundation for high-level human-machine collaboration.

[0069] In any embodiment of this application, the method for determining the prompt information includes, but is not limited to, at least one of the following: in response to display constraints including spatial size parameters, determining the content indicated by the prompt information based on the relationship between the spatial size parameters and the corresponding size threshold; in response to display constraints including device movement status, determining the content indicated by the prompt information based on the relationship between the movement speed in the device movement status and the corresponding speed threshold; in response to display constraints including historical interaction behavior, determining the interaction activity level of the status area based on the historical interaction behavior, and determining the content indicated by the prompt information based on the relationship between the interaction activity level and the corresponding activity level threshold.

[0070] Interaction activity includes, but is not limited to, click frequency, dwell time, number of operations, etc. For example, interaction activity can be calculated based on at least one of the following: the number of times the user clicks on the status area, the duration of long press, the duration of gaze, or the frequency of swipe operations within a preset time period.

[0071] Among them, the size threshold corresponding to each spatial size parameter can be preset, and the size threshold corresponding to different types of spatial size parameters can be different. For example, the size threshold corresponding to the area ratio is different from the size threshold corresponding to the area size.

[0072] The activity threshold for each type of interaction activity can be preset, and the activity thresholds for different types of interaction activity can be different. For example, the activity threshold for click frequency is different from the activity threshold for dwell time (or gaze duration).

[0073] Understandably, by dynamically determining the granularity of prompt information using a threshold mechanism corresponding to multi-dimensional display constraints, adaptive optimization of information presentation can be achieved. For example, when there is sufficient interface space in the status area, the device is moving at low speed or stationary, or the user shows high interactive activity in the status area, the system can provide the agent's status and rich contextual information, enhancing task transparency and user control. Conversely, when interface space in the status area is limited, the device is moving at high speed, or the user shows low interactive activity in the status area, only core status information is displayed, effectively avoiding visual interference and cognitive overload. Thus, both information completeness and interface simplicity can be balanced, significantly improving the intelligence, security, and consistency of user experience in human-computer interaction.

[0074] As an example, the content indicated by the prompt message is determined to include both the state of the agent and the subtask description information at the current time step in the execution path associated with the interactive task, provided that any of the following conditions are met: the spatial size parameter of the state area is greater than or equal to the corresponding size threshold; the device moving speed is less than or equal to the speed threshold; or the interactive activity level of the state area is greater than or equal to the corresponding activity threshold.

[0075] The execution path is generated by the agent based on the task instructions and the screen content of the user interface, and includes subtask description information at at least one time step.

[0076] As another example, the content indicated by the prompt message is determined to include the state of the agent if any of the following conditions are met: the spatial size parameter of the state area is less than the corresponding size threshold; the device movement speed is greater than the speed threshold; or the interaction activity of the state area is less than the corresponding activity threshold. For example, when the prompt message is presented in a static format, the content indicated by the prompt message may only include the state of the agent; while when the prompt message is presented in a dynamic format (such as scrolling display, page turning display, etc.), the content indicated by the prompt message may simultaneously include the state of the agent and the subtask description information at the current time step in the execution path associated with the interactive task.

[0077] In summary, based on the display constraints of the status area in the user interface, the granularity of the prompt information is dynamically adjusted. When the status area is relatively small, the device movement speed is relatively high, and the user interaction is infrequent, the current state of the agent is displayed, ensuring a concise and focused interface. When the status area is relatively large, the device movement speed is relatively low, and the user interaction is frequent, subtask descriptions at the current time step are overlaid, providing richer task context. This allows for full utilization of limited interface space resources, adaptively balancing information density and readability without interfering with user operations. It ensures immediate awareness of critical states and, when conditions permit, enhances the user's understanding of the task execution path, thereby improving the efficiency and consistency of human-computer collaboration.

[0078] In any embodiment of this application, the operation area may include an application window of the target application. In this application, the application window may display operation information when the intelligent agent performs a target operation on the target application. The target operation is the operation associated with the subtask description information at the current time step in the execution path associated with the interactive task; the execution path is generated by the intelligent agent based on the screen content of the task instruction user interface.

[0079] In any embodiment of this application, the operation area may also include a mask window surrounding the application window. In this application, interactive animations when the intelligent agent performs a target operation on the target application can be displayed in the mask window.

[0080] The target operation is the operation associated with the subtask description information at the current time step in the execution path associated with the interactive task; the execution path is generated by the agent based on the screen content of the user interface of the task instruction.

[0081] Understandably, the application window presents specific operational semantics, allowing users to clearly understand the agent's behavioral intent; the overlay window, with non-intrusive visual animation, is superimposed on the original interface, intuitively indicating the location and temporal rhythm of the operation. The two work together to achieve the separation and fusion of semantic readability and visual guidance, which not only avoids the operation prompts from obscuring the original interface content, but also enhances the synchronous perception of human-computer operation. Without interfering with the normal display of the target application, it significantly improves the transparency, interpretability, and user control of the interaction process.

[0082] Non-intrusive refers to interactive effects presented as overlays, without modifying, obscuring, or interrupting the original interface content and functions of the target application. Users can clearly observe the original interface content and operation prompts of the target application at the same time, thereby achieving low-interference and high-transparency interactive feedback.

[0083] The interaction method of this application embodiment, after receiving a task instruction, calls an intelligent agent to perform an interactive task that matches the semantics of the task instruction with the current screen content, and simultaneously displays at least one of the following in the user interface: operation information, interactive animation effects, and prompt information matching the state of the intelligent agent: the operation information intuitively presents the specific behavior of the intelligent agent on the target application, the interactive animation effects visually enhance the spatiotemporal correlation of the operation, and the prompt information reflects the actual state (or execution stage) of the intelligent agent in real time, thereby constructing a closed-loop interactive feedback mechanism that is "visible, knowable, and perceptible", significantly improving the user's understanding and control of the intelligent agent's behavior, and enhancing the naturalness, transparency, and user experience of human-computer collaboration while ensuring the accuracy of task execution.

[0084] As one possible implementation method, Figure 3 A flowchart illustrating yet another interaction method provided for an exemplary embodiment of this application.

[0085] It should be noted that this interaction method can be executed alone, or it can be executed together with any embodiment or possible implementation in the embodiment of this application, or it can be executed together with any technical solution in the related technology. The embodiments of this application do not limit this.

[0086] like Figure 3 As shown, the interaction method may include the following steps S301 to S304: Step S301: In response to receiving a task instruction, the agent is invoked to execute an interactive task matching the task instruction based on the task instruction and the screen content of the user interface.

[0087] It should be noted that the explanation of step S301 can be found in the relevant description in any embodiment of this application, and will not be repeated here.

[0088] Step S302: During the execution of the interaction task, in response to receiving a first target event that conflicts with the actual state of the agent, determine the conflict detection type to which the first target event belongs.

[0089] The collision detection types include, but are not limited to: occlusion detection, touch detection, and incoming call detection.

[0090] In this embodiment, when a first target event that conflicts with the actual state of the agent is detected, the conflict detection type of the first target event can be determined. As one possible implementation, different associations between event signal values ​​and conflict detection types can be pre-registered. Therefore, in this application, the aforementioned associations can be queried based on the event signal value of the first target event to determine the conflict detection type of the first target event.

[0091] Step S303: Determine the first target state that matches the conflict detection type from the agent's state machine.

[0092] The state machine includes various states that the agent may be in, including but not limited to: initial state, running state (active state), paused state (temporary waiting state), stopped state, and completed state (end state).

[0093] The first target state includes, but is not limited to: a paused state used to instruct the agent to suspend execution, and a stopped state used to instruct the agent to stop execution.

[0094] In this embodiment, a first target state matching the conflict detection type of the first target event can be determined from the state machine of the agent. As a possible implementation, the matching relationships between different states and conflict detection types can be pre-configured. Therefore, in this application, the matching relationships can be queried based on the conflict detection type of the first target event to determine the first target state matching the conflict detection type of the first target event.

[0095] In one possible implementation of the embodiments of this application, the first target event associated with the paused state includes, but is not limited to, at least one of the following: The first item: Size change events and / or position change events of the application window of the target application associated with the interactive task; wherein, the application window is used to display the agent's operation information on the target application.

[0096] For example, a user can trigger an application window position change event by dragging the application window; a user can trigger an application window size change event by scaling the application window using a two-finger pinch gesture.

[0097] The second item: Target occlusion event when the application window switches from an unoccluded state to an occluded state. For example, alarm pop-ups, notification pop-ups, etc., occluding the application window will trigger the target occlusion event.

[0098] As one possible implementation, when the executing subject in this application embodiment is a vehicle, the target occlusion event does not include the occlusion event of the turn signal pop-up. Thus, the execution efficiency and success rate of the interactive task can be improved without affecting the safe driving of the vehicle.

[0099] The third item: the state transition event when the target application switches from a foreground state to a background state.

[0100] Fourth item: Incoming call incidents.

[0101] Fifth: In response to a task instruction being a voice instruction, a new received voice interaction event is added; this voice interaction event is used to pause the execution of the interaction task. For example, while the agent is operating the target application, the user can pause or stop the execution of the interaction task by voice inputting another valid question.

[0102] Item 6: Pause events sent from the cloud.

[0103] For example, in a scenario where the intelligent agent is deployed in the cloud, the task scheduling module in the cloud can issue a pause event to the electronic device according to a policy or user request to interrupt its ongoing interactive operation.

[0104] For example, in a scenario where the intelligent agent is deployed on an electronic device (such as a vehicle), the user can send a pause event to the vehicle through the terminal application (APP) to instruct it to pause the current interactive task (such as music playback).

[0105] Item 7: State switching event where the voice interaction function changes from an available state to an unavailable state.

[0106] Item 8: Screen state transition event from on screen to locked screen.

[0107] Item 9: Screen state transition event from on-screen to off-screen.

[0108] Item 10: Pause events initiated by the user on interactive controls in the status area (such as the pause button).

[0109] Item 11: Screen touch events, that is, events triggered by the user touching the screen.

[0110] Item 12: Touch events initiated by the user on the overlay window in the user interface's operation area; where the overlay window is used to display interactive animations, which are associated with the operations performed by the agent on the target application associated with the interactive task.

[0111] As one possible implementation, taking a screen projection scenario as an example, assuming that the execution subject of this application embodiment is a vehicle and the user interface is projected from the terminal onto the vehicle, the first target event associated with the paused state may also include: a pop-up reminder event when the terminal projects its screen onto the vehicle for the first time.

[0112] In summary, when the first target state is in a paused state, the first target event can cover a variety of triggering events, that is, it supports the association between multiple events and the paused state, thereby flexibly driving the state switching of the intelligent agent and better adapting to the personalized needs of different application scenarios.

[0113] In one possible implementation of the embodiments of this application, the first target event associated with the stopped state includes, but is not limited to, at least one of the following: The first item is the stop event sent from the cloud.

[0114] The second item is to add a received voice-based user interaction event in response to the task instruction being a voice instruction; wherein the voice-based user interaction event is used to stop the execution of the interaction task and / or execute a new interaction task.

[0115] For example, the voice-based user interaction event is used to trigger the agent to stop performing the interaction task, so as to invoke the agent (such as the current agent or other agents) to perform a new interaction task that matches the voice-based user interaction event.

[0116] The third item is a stop event initiated by the user on an interactive control (such as a stop button) in the status area of ​​the user interface; where the status area is used to display prompt information that matches the state of the agent.

[0117] Item 4: In response to a task instruction being a voice instruction, add received non-voice user interaction events.

[0118] As one possible implementation, taking a screen projection scenario as an example, assuming that the execution subject of this application embodiment is a vehicle and the user interface is projected from the terminal onto the vehicle, the first target event associated with the stationary state may also include: a connection disconnection event between the terminal and the vehicle.

[0119] In summary, when the first target state is a stopped state, the first target event can cover a variety of triggering events, that is, it supports the association between multiple events and the stopped state, thereby flexibly driving the state switching of the intelligent agent and better adapting to the personalized needs of different application scenarios.

[0120] Step S304: Update the state of the agent to the first target state.

[0121] It should be noted that the explanation of step S304 can be found in the relevant description in any embodiment of this application, and will not be repeated here.

[0122] In any embodiment of this application, when the first target state is a stopped state, the agent can be controlled to stop performing the interactive task. For example, when the agent is deployed on the electronic device side, the electronic device can directly control the agent to stop performing the interactive task; while when the agent is deployed on the cloud side, the electronic device can report the first target event or the stopped state to the cloud so that the cloud controls the agent to stop performing the interactive task.

[0123] In any embodiment of this application, when the first target state is a stopped state, at least one of the following can also be performed: The first item: Hide the state area of ​​the user interface; the state area is used to display prompts that match the state of the agent.

[0124] The second item: Hide the overlay window in the operation area of ​​the user interface; the overlay window is used to display interactive animations, which are associated with the operations performed by the agent on the target application associated with the interactive task.

[0125] In any embodiment of this application, when the first target state is a paused state, the agent can be controlled to pause the execution of interactive tasks, and / or, a prompt message matching the paused state can be displayed in the status area of ​​the user interface. Thus, while ensuring the robustness of the agent's operation, timely and consistent status feedback enhances the predictability and transparency of human-machine collaboration, effectively avoiding task execution deviations or user confusion.

[0126] For example, when the agent is deployed on the electronic device side, the electronic device can directly control the agent to pause the execution of interactive tasks; while when the agent is deployed on the cloud side, the electronic device can report the first target event or pause state to the cloud so that the cloud can control the agent to pause the execution of interactive tasks.

[0127] The interaction method in this application embodiment automatically identifies the conflict detection type of a first target event that conflicts with the agent's current actual state when it receives such an event. It then determines a first target state matching the conflict detection type from predefined states in the state machine and updates the agent's state to the first target state, thus achieving a structured response to conflict events. This allows the agent to process concurrent or abnormal events in an orderly manner according to predefined state transition rules, ensuring the robustness of the agent's operation.

[0128] As one possible implementation method, Figure 4 A flowchart illustrating another interaction method provided for an exemplary embodiment of this application.

[0129] It should be noted that this interaction method can be executed alone, or it can be executed together with any embodiment or possible implementation in the embodiment of this application, or it can be executed together with any technical solution in the related technology. The embodiments of this application do not limit this.

[0130] like Figure 4 As shown, the interaction method may include the following steps S401 to S407: Step S401: In response to receiving a task instruction, the agent is invoked to execute an interactive task matching the task instruction based on the task instruction and the screen content of the user interface.

[0131] Step S402: During the execution of the interactive task, in response to receiving a first target event that conflicts with the agent's actual state, a first target state that matches the first target event is determined, and the state of the agent is updated to the first target state.

[0132] Step S403: In response to the first target state being in a paused state, control the intelligent agent to pause the execution of the interactive task.

[0133] It should be noted that the explanations of steps S401 to S403 can be found in the relevant descriptions in any embodiment of this application, and will not be repeated here.

[0134] Step S404: Determine the recovery strategy associated with the operation source of the first target event; wherein the recovery strategy is used to indicate the allowed duration of the paused state and / or the countdown duration.

[0135] The sources of operations include, but are not limited to: cloud operations (such as cloud-initiated status updates such as pause, stop, and complete), system event operations (such as interface obstruction, incoming calls, and application window changes), user click operations, and special behavior operations (such as user touch screens).

[0136] The recovery strategy associated with each operation source can be pre-set, with different recovery strategies associated with different operation sources, or the recovery strategies associated with multiple operation sources can not be completely the same.

[0137] As an example, for cloud operations or system event operations, the recovery strategy can be to resume after waiting for a first set duration; wherein, the first set duration can be controlled by the cloud, and for example, the first set duration is 8 seconds (s); for special behavior operations, the recovery strategy can be to resume after pausing for a first set duration plus counting down for a second set duration; wherein, the second set duration can be controlled by the cloud, and this application does not limit the relationship between the first set duration and the second set duration, for example, both durations can be 8 seconds.

[0138] In the embodiments of this application, the operation source of the first target event can be determined, and the recovery strategy associated with the operation source can be determined.

[0139] Step S405: In response to the fact that the duration of the agent's paused state has not reached the duration indicated by the recovery strategy, and a second target event that conflicts with the first target state is received, a second target state that matches the second target event is determined from the agent's state machine.

[0140] The second target state includes, but is not limited to: the execution state, which instructs the agent to continue execution, and the stop state, which instructs the agent to stop execution.

[0141] In this embodiment, if the duration of the agent's paused state does not reach the duration indicated by the recovery strategy, it can be further determined whether a second target event conflicting with the first target state has been received. If so, a second target state matching the second target event can be determined from the agent's state machine. For example, a second target state matching the conflict detection type of the second target event can be determined from the agent's state machine.

[0142] In one possible implementation of the embodiments of this application, the second target event associated with the stopping state includes, but is not limited to, at least one of the following: The first item is the stop event sent from the cloud.

[0143] The second item is to add a received voice-based user interaction event in response to the task instruction being a voice instruction; wherein the voice-based user interaction event is used to stop the execution of the interaction task and / or execute a new interaction task.

[0144] For example, the voice-based user interaction event is used to trigger the agent to stop performing the interaction task, so as to invoke the agent (such as the current agent or other agents) to perform a new interaction task that matches the voice-based user interaction event.

[0145] The third item is a stop event initiated by the user on an interactive control (such as a stop button) in the status area of ​​the user interface; where the status area is used to display prompt information that matches the state of the agent.

[0146] Item 4: In response to a task instruction being a voice instruction, add received non-voice user interaction events.

[0147] As one possible implementation, taking a screen projection scenario as an example, assuming that the execution subject of this application embodiment is a vehicle and the user interface is projected from the terminal onto the vehicle, the second target event associated with the stationary state may also include: a connection disconnection event between the terminal and the vehicle.

[0148] In one possible implementation of the embodiments of this application, the second target event associated with the execution state includes, but is not limited to, at least one of the following: First item: Call termination event.

[0149] The second item: Continue events initiated by the user on interactive controls (such as the continue button) in the status area of ​​the user interface; where the status area is used to display prompt information that matches the state of the agent.

[0150] The third item: A click event by the user on the target option among multiple candidate options associated with the rhetorical question sent by the agent in the user interface's operation area. Here, the multiple candidate options are the options the agent needs to decide on, and the operation area is used to indicate the agent's operation information and / or interactive animations related to the target application of the interactive task.

[0151] For example, taking an agent deployed in the cloud as an example, in a cloud-based questioning scenario, if a user clicks on an option on the screen, the agent is instructed to continue performing the interactive task. For instance, suppose the interactive task is to buy movie tickets, and the application window in the operation area displays movie tickets for multiple time slots. The cloud can ask the user which time slot they want to buy. If the user clicks on a movie ticket for one of the time slots, the agent can continue performing the movie ticket purchase task based on the selected ticket.

[0152] The fourth item: The recovery event when the application window of the target application in the operation area is restored from an obscured state to an unobscured state; where the application window is used to display operation information.

[0153] Fifth item: State transition event when the target application switches from background to foreground state.

[0154] Item 6: Screen state transition event from locked screen to on screen.

[0155] Item 7: Screen state transition event from off state to on state.

[0156] Item 8: State switching event where the voice interaction function is restored from an unavailable state to an available state.

[0157] In summary, when the second target state is the execution state, the second target event can cover a variety of triggering events, that is, it supports the association between multiple events and the execution state, thereby flexibly driving the state switching of the intelligent agent and better adapting to the personalized needs of different application scenarios.

[0158] Step S406: Update the state of the agent to the second target state.

[0159] As an example, when the agent is deployed on an electronic device, the electronic device can directly update the state of the agent to the second target state and display a prompt message matching the second target state in the state area of ​​the user interface.

[0160] As another example, when the agent is deployed in the cloud, the electronic device can report a second target event or a second target state to the cloud, so that the cloud updates the agent's state to the second target state. Simultaneously, the electronic device can also display a prompt message matching the second target state in the user interface's status area.

[0161] In any embodiment of this application, when the second target state is in the execution state, the agent can also be controlled to resume executing the interactive task.

[0162] In any embodiment of this application, when the second target state is the execution state, at least one of the following can also be executed: The first item: In the operation area of ​​the user interface, continue to display the operation data of the intelligent agent on the target application.

[0163] The second item: In the operation area of ​​the user interface, continue to display interactive animations associated with the operations performed by the agent on the target application.

[0164] The third item: Display prompts in the status area of ​​the user interface that match the execution status.

[0165] In any embodiment of this application, when the second target state is in a stopped state, the intelligent agent can also be controlled to stop performing interactive tasks.

[0166] In any embodiment of this application, when the second target state is in a stopped state, at least one of the following can also be performed: Item 1: Hide the state area of ​​the user interface; where the state area is used to display prompts that match the state of the agent.

[0167] Item 2: Hide the overlay window in the operation area of ​​the user interface; the overlay window is used to display interactive animations, which are associated with the operations performed by the agent on the target application associated with the interactive task.

[0168] In summary, the agent's behavior and interface feedback are dynamically adjusted based on the second target state: when resuming the execution state, the agent seamlessly continues to execute the interactive task and continuously displays operation data and / or interactive animations in the operation area, ensuring the user's continuous perception of the task progress; when switching to the stopped state, the task is actively terminated and the overlay window in the state area and operation area is hidden, releasing interface resources in a timely manner and eliminating redundant visual interference. This achieves synergistic consistency between behavioral logic and interface presentation, ensuring both the recoverability and contextual coherence of task execution, while avoiding interface clutter caused by invalid information residue, significantly improving the clarity, responsiveness, and user experience of human-computer interaction.

[0169] In step S407, in response to the duration reaching the duration indicated by the recovery strategy and no second target event being received, the state of the agent is updated to the execution state to control the agent to resume the execution of the interactive task.

[0170] In this embodiment of the application, if the duration of the agent in the paused state reaches the duration indicated by the recovery strategy, it can be determined whether a second target event that conflicts with the first target state has been received. If not, the state of the agent is updated to the execution state so as to control the agent to resume the execution of the interactive task.

[0171] In any embodiment of this application, when the state of the agent is updated to the execution state, at least one of the following can also be displayed synchronously in the user interface: in the operation area of ​​the user interface, the operation information of the agent on the target application is continued to be displayed; in the operation area of ​​the user interface, the interactive animation associated with the operation performed by the agent on the target application is continued to be displayed; in the state area of ​​the user interface, prompt information matching the execution state is displayed.

[0172] Understandably, when an agent is in a suspended state for a relatively long period without any new conflict events, automatically and orderly resuming task execution and synchronously updating the interface feedback can ensure the user's immediate awareness of task continuation and contextual consistency. Thus, achieving timeout self-recovery and consistent interface state coordination without user intervention avoids task failure due to prolonged suspension while maintaining the smoothness, transparency, and integrity of human-computer interaction.

[0173] It should be noted that steps S405 to S406 and step S407 are two possible implementations in parallel, and either one can be selected for execution in actual application.

[0174] The interaction method of this application embodiment dynamically determines the recovery strategy (such as the allowed pause duration and / or countdown duration) based on the operation source of the first target event when the agent enters the paused state, and continuously listens for or receives new events during this period. When a second target event that conflicts with the current paused state is detected or received, the agent quickly switches to the second target state that matches the second target event based on a predefined state machine. This not only achieves context-aware management of pause behavior (avoiding unlimited suspension), but also supports flexible response to conflict events during the pause process, ensuring the rationality and timeliness of the agent's state transition. While improving system robustness, it also considers the flexibility of interaction and the consistency of user experience, effectively preventing task stagnation or user operation defocusing caused by rigid pauses.

[0175] As one possible implementation method, Figure 5 A flowchart illustrating another interaction method provided for an exemplary embodiment of this application.

[0176] It should be noted that this interaction method can be executed alone, or it can be executed together with any embodiment or possible implementation in the embodiment of this application, or it can be executed together with any technical solution in the related technology. The embodiments of this application do not limit this.

[0177] like Figure 5 As shown, the interaction method may include the following steps S501 to S503: Step S501: In response to receiving a task instruction, the agent is invoked to execute an interactive task matching the task instruction based on the task instruction and the screen content of the user interface.

[0178] It should be noted that the explanation of step S501 can be found in the relevant description in any embodiment of this application, and will not be repeated here.

[0179] Step S502: In response to the agent completing the interactive task, update the state of the agent to the completed state, and / or hide the state area in the user interface, and / or hide the overlay window in the operation area of ​​the user interface.

[0180] The status area displays prompts that match the agent's current state; the overlay window displays interactive animations, which are associated with the actions performed by the agent on the target application related to the interactive task.

[0181] In this embodiment of the application, when the agent completes the interactive task, at least one of the following can be performed: automatically updating the state of the agent to the completed state; hiding the state area of ​​the user interface; hiding the overlay window in the operation area; wherein the overlay window is used to display interactive animations associated with the operation performed by the agent on the target application.

[0182] Step S503: In response to the state of the agent being updated to the completed state, clear the agent's executed operations and cached data.

[0183] In this embodiment, when the state of the agent is updated to the completed state, the executed operations and cached data of the agent can also be cleared. This helps to release system resources, avoid redundant information from interfering with subsequent tasks, and ensure that the agent is in a clean and consistent initial state in the next run, thereby improving the reliability of task execution and system stability.

[0184] In any embodiment of this application, when the state of the agent is updated to the stopped state, the agent's executed operations and cached data can also be cleared to further reduce resource consumption.

[0185] The interaction method of this application embodiment automatically updates the state of the intelligent agent to the completed state after the intelligent agent completes the interaction task, and / or synchronously hides the state area in the user interface, and / or synchronously hides the overlay window used to display interactive animation effects in the operation area of ​​the user interface, thereby releasing interface resources in a timely manner and eliminating redundant visual elements. On the one hand, it avoids the interference of prompt information or residual interactive animation effects of the completed task with the user's subsequent operation; on the other hand, through strict synchronization between the state and the interface performance, it ensures that the user clearly perceives that the task has been completed, improving the clarity of the interaction loop and the cleanliness of the interface.

[0186] In any embodiment of this application, taking a screenshot as an example, the application can be automatically controlled to complete interactive tasks by combining application screenshot understanding and large model technology through pause, continue, and question functions. For example, a multi-terminal universal, unified, simple, and reusable UI Agent software development kit (SDK) can be integrated into electronic devices. By constructing three modules—state management, conflict management, and UI management—the interaction of the UI Agent universal framework can be realized.

[0187] As an example, taking electronic devices such as mobile phones or in-vehicle systems, with the UI Agent deployed in the cloud, the interaction system architecture can be as follows: Figure 6 The UI Agent SDK Manager is the core of the entire interaction system, responsible for instruction parsing and state management, and coordinating the work between various modules. Instruction parsing is responsible for parsing operation instructions from mobile / vehicle clients, including operations such as start, pause, complete, stop, and restart. The state management module is responsible for maintaining the overall state of the UI Agent, including initial state, running state, paused state, stopped state, and completed state. State area and overlay implementation injection: The state management module transmits state information to the UI management module so that the agent's current state can be displayed on the user interface in real time. Listeners: The state management module receives state change requests from other modules, such as pause and stop, through listeners. The conflict management module is responsible for detecting and handling internal and external conflicts. The UI management module is responsible for the unified management of the overlay window, state area window, and state distribution. The UI Agent data storage module is responsible for data storage (storing various data of the UI Agent during operation, including state change records, conflict event records, user operation records, agent operation records, etc.) and data retrieval (when it is necessary to restore the state or query historical records, other modules can read the corresponding data from the UI Agent data storage module).

[0188] against Figure 6The arrows in the diagram indicate the following: UI Agent SDK Manager -> Command Parsing: The UI Agent SDK Manager passes the received operation commands to the command parsing module for parsing; UI Agent SDK Manager -> State Area and Mask Implementation Injection -> UI Management: The UI Agent SDK Manager injects the predefined state area and mask implementation logic into the UI management module. The state area displays the agent's current state, and the mask displays interactive animations related to the operation performed by the agent on the target application, overlaying it on the application interface to attract user attention or prevent other operations. Through this injection method, the UI management module can accurately display the state area and mask when needed based on the injected implementation logic, thereby improving the user interface's interactive experience and information delivery effect; Command Parsing -> State Management and Conflict Management: The parsed operation commands are passed to the state management and conflict management modules to execute the corresponding operations; State Management <- Listener: The state management module receives state change requests from other modules through listeners; State Management -> UI Agent Data Storage: The state management module passes important state change information to the UI Agent. The Agent data storage module stores the data; Conflict Management -> State Management: After detecting a state conflict, the conflict management module sends the corresponding event signal to the state management module, triggering a state change for the agent; Conflict Management -> UI Agent Data Storage: The conflict management module transmits the conflict event and related state change information to the UI Agent data storage module for recording; State Management -> UI Management: The state management module drives the UI management module to adjust the user interface display. For example, when a conflict event is received, it will cause a change in the state in the state management module. At this time, the state management module can notify the UI management module to adjust the user interface display according to the new state; UI Management -> UI Agent Data Storage: The UI management module transmits user operation records, agent operation records, and interface state information to the UI Agent data storage module for saving.

[0189] The following will explain in detail. Figure 6 It includes several main parts: 1. Status Management The client maintains the overall state of the UI Agent uniformly, and a state machine is designed to implement state management. The state machine can include, for example,... Figure 7 The five states shown are uniformly maintained for state switching, ensuring reusability across multiple devices: Initial state: The initial state of the UI Agent, which involves parsing cloud information, recording the original ID, and initializing UI Agent operations; Execution state (active state): Maintains the state of the UI Agent in the state area and keeps track of the UI Agent's execution progress; Paused State: This is a temporary waiting state triggered by user-initiated pause or interface obstruction. It actively sends a PauseEvent to the cloud, ending the current round of requests. In the paused state, if the environment returns to normal after a period of time (e.g., the obstruction is gone), it can automatically resume the running state, actively sending a ResumeEvent to the cloud. If the environment has not returned to normal after a period of time (e.g., the obstruction is still present), the interactive task can automatically end, avoiding prolonged waiting. Stopped state: A state that cannot be resumed after a period of time due to user-initiated termination or pause. It ends the state area in the user interface and clears the execution operations and cached data of the UI Agent. Completed State (Ended State): The state after the UI Agent completes the interactive task. It ends the state area in the user interface and clears the UI Agent's execution operations and cached data.

[0190] 2. State switching Four operation sources plus N system event operations cater to various customized scenarios and enable flexible state switching. Furthermore, different flags can be used to quickly trigger and resume system event operations. Cloud-based operation: The cloud proactively sends status updates such as pause, stop, and complete. System event operations: Current screen interface obstruction, incoming call, application window changes, other valid queries and other interfering events; User click actions: Users can actively click on the status area to initiate pause, stop, or resume; Special actions: Special actions such as user touch on the screen.

[0191] Regardless of the number of special scenarios on each terminal, different types of countdown polling can be used to resume or exit the system through different operation sources: For cloud operations and system event operations, the recovery strategy can be to wait for a first set time before resuming; wherein, the first set time can be controlled by the cloud, for example, the first set time is 8 seconds (s). For special behavior operations, the recovery strategy can be to pause for a first set duration + countdown for a second set duration and then resume; wherein, the second set duration can be controlled by the cloud, and this application does not limit the relationship between the first set duration and the second set duration. For example, both durations can be 8 seconds. For user clicks in cloud-based query scenarios, the recovery strategy can be immediate recovery.

[0192] For example, the relationship between system event operations, operation sources, and recovery strategies can be as follows: Figure 8 As shown.

[0193] In this application, the UI Agent SDK maintains a set of rule logic for managing the UI Agent's state, providing users with a smooth UI experience. The cloud can issue commands according to these rules to facilitate UI Agent state switching, primarily addressing cloud-based state switching in state management and internal conflict management.

[0194] 3. Conflict Management Conflicts can be categorized based on whether they arise from one's own business operations, such as... Figure 9 And the internal and external conflicts shown in Table 1.

[0195] Table 1 Internal and External Conflicts

[0196] in, Figure 9 The "Page Occlusion," "Application Window Position Change," and "Screen Lock" functions are used to trigger the agent's state update to a paused state; "Page Visibility" is used to trigger the agent to resume execution from a paused state when the page is visible; "Wake-up" is used to wake up the device when a user's voice input wake-up command is detected; "Command Parsing" is used to parse the user's voice input task commands; and "Multi-round UI Agent Conflict" refers to the situation where, if a new task command is received during UI Agent operation, the UI Agent is controlled to stop executing the current interaction task in order to control other UI Agents. The Agent executes new interactive tasks; "Valid query" is used to detect valid task instructions input by the user; "Interruption callback (onInterrupt)" is used to receive interrupt signals from the system or external events and control the agent to interrupt the execution of the current interactive task; "Resume callback (onResume)" is used to control the agent to resume the execution of the interrupted interactive task; "pkgName (package name resolution)" is used to locate the target application associated with the interactive task; "Window boundary resolution (windowBounds)" is used to resolve the boundary coordinate information of the application window (such as the top left corner position, width, and height) to determine its visible range and relative layout on the screen, thereby supporting the agent to accurately locate interactive elements (such as buttons and input boxes) and avoid accidentally touching other application or system controls, improving the accuracy and robustness of operation.

[0197] Conflict management provides event signals for automatic recovery, which the automatic recovery module detects and changes the state of UIAgent. For example, the state change principle can be as follows: Figure 10 As shown, it mainly includes the following steps: 1) The application layer registers conflict detection types with the UI Agent SDK, such as occlusion detection, foreground / background changes, and application size changes; 2) During the execution of the UI Agent, the application layer sends the corresponding event signals to the UI Agent SDK; 3) The UI Agent SDK calls back to the state management module to change the state based on the signal value of the event signal; 4) The status change trigger operation area and status area are displayed accordingly.

[0198] 4. UI Management The client-side interface features a unified overlay window, status area window, and status distribution module; each client can customize the different parts. Window management: Based on the state changes of the UI Agent, manage the initialization, creation, size, position adjustment, display, and hiding of windows; Overlay Window: Light Effect Display: Displays light effect animations and click effects during UI Agent execution; Operation Interception: Intercepts user operations and transmits UI Agent operations; State Distribution: State changes come from the state management module, used to manage interface light effects and display them to the user.

[0199] State Area: The UI Agent SDK provides an abstract class, and the business logic is implemented by each client. The state area is used to collect operation events (such as the user actively clicking the pause, stop, and continue buttons in the state area), which are recorded and scheduled by the UI Agent SDK. State Distribution: State changes come from the state management module, which is used to manage the state changes and text changes of the UI Agent in the state area and display them to the user.

[0200] For example, the system architecture can be as follows Figure 11 As shown, the main functions include: when the task starts, the overlay window displays light effects animations and the status area displays text; during task execution, there may be status changes (e.g., when the user clicks pause, the light effects in the overlay window and the text in the status area need to be adjusted) and operation events (intercepting user operations and passing through the UI Agent's click, swipe, and other operations); when the task ends, the overlay window and status area are hidden.

[0201] In summary, the solution provided in this application has at least the following advantages: it can quickly support UI Agent capabilities on multiple devices such as in-vehicle systems and mobile phones; it enriches user voice operation application scenarios, improves interaction efficiency, and enhances the automation of complex interaction tasks.

[0202] To implement the above embodiments, this application also proposes an interactive device.

[0203] Figure 12This is a schematic diagram of the structure of an interactive device provided for an exemplary embodiment of this application.

[0204] like Figure 12 As shown, the interactive device 1200 may include: an execution module 1210, a determination module 1220, and an update module 1230.

[0205] The execution module 1210 is used to respond to receiving a task instruction by calling the intelligent agent to perform an interactive task matching the task instruction based on the task instruction and the screen content of the user interface. The determination module 1220 is used to determine a first target state that matches the first target event in response to receiving a first target event that conflicts with the actual state of the agent during the execution of the interactive task. Update module 1230 is used to update the state of the agent to the first target state.

[0206] In one implementation of this application, the user interface includes an operation area and a status area, and the interactive device 1200 further includes: The display module is used to synchronously display at least one of the following during the execution of the interactive task: in the operation area, displaying the operation information of the agent on the target application associated with the interactive task; in the operation area, displaying interactive animations, wherein the interactive animations are associated with the operations performed by the agent on the target application; in the status area, displaying prompt information matching the status of the agent.

[0207] In one implementation of this application, the display module is specifically used to: determine the content indicated by the prompt message according to the display constraints corresponding to the status area; wherein, the display constraints include at least one of the following: the spatial size parameters of the status area in the user interface, the device movement status, and the historical interaction behavior associated with the status area; and display the content indicated by the prompt message in the status area.

[0208] In one implementation of this application, the display module is specifically configured to perform at least one of the following: in response to display constraints including spatial size parameters, determining the content indicated by the prompt message based on the relationship between the spatial size parameters and the corresponding size threshold; in response to display constraints including device movement status, determining the content indicated by the prompt message based on the relationship between the movement speed in the device movement status and the corresponding speed threshold; in response to display constraints including historical interaction behavior, determining the interaction activity level of the status area based on the historical interaction behavior, and determining the content indicated by the prompt message based on the relationship between the interaction activity level and the corresponding activity level threshold.

[0209] In one implementation of this application, the display module is specifically used to: determine, in response to satisfying any of the following, that the content indicated by the prompt message includes the state of the agent: the spatial size parameter is less than the corresponding size threshold; the movement speed is greater than the speed threshold; and the interaction activity is less than the corresponding activity threshold.

[0210] In one implementation of this application, the display module is specifically used to: determine, in response to satisfying any one of the following, that the content indicated by the prompt message includes the state of the agent and the subtask description information at the current time step in the execution path associated with the interaction task: the spatial size parameter is greater than or equal to the corresponding size threshold; the movement speed is less than or equal to the speed threshold; and the interaction activity is greater than or equal to the corresponding activity threshold. The execution path is generated by the agent based on the task instructions and the screen content.

[0211] In one implementation of this application, the operation area includes an application window of the target application; the display module is specifically used to: display operation information of the intelligent agent when performing the target operation on the target application in the application window; wherein, the target operation is the operation associated with the subtask description information at the current time step in the execution path associated with the interactive task; the execution path is generated by the intelligent agent according to the task instructions and the screen content.

[0212] In one implementation of this application, the operation area further includes a mask window surrounding the application window; the display module is further configured to: display interactive animations of the intelligent agent performing the target operation on the target application in the mask window.

[0213] In one implementation of this application, the determining module 1220 is specifically used to: in response to receiving a first target event that conflicts with the actual state, determine the conflict detection type to which the first target event belongs; and determine a first target state that matches the conflict detection type from the state machine of the agent.

[0214] In one implementation of this application embodiment, in response to the first target state being a paused state for controlling the agent to suspend the execution of interactive tasks, the interactive device 1200 may further include: The first processing module is used to determine a recovery strategy associated with the operation source of the first target event; wherein the recovery strategy is used to indicate the allowed duration of the paused state and / or the countdown duration; in response to the agent being in the paused state for a duration that does not reach the duration indicated by the recovery strategy, and receiving a second target event that conflicts with the first target state, the module determines a second target state that matches the second target event from the agent's state machine; and updates the state of the agent to the second target state.

[0215] In one implementation of this application embodiment, in response to the second target state being an execution state for controlling the agent to resume performing the interaction task, the interaction device 1200 may further include: The second processing module is configured to perform at least one of the following: continue to display the agent's operation data on the target application in the operation area of ​​the user interface; continue to display interactive animations in the operation area of ​​the user interface, wherein the interactive animations are associated with the operations performed by the agent on the target application; and display prompt information matching the execution state in the status area of ​​the user interface.

[0216] In one implementation of this application, in response to a first target state or a second target state being a stopped state for controlling the agent to stop performing an interactive task, the interactive device 1200 may further include: The third processing module is used to perform at least one of the following: hide the state area in the user interface; wherein the state area is used to display prompt information matching the state of the agent; hide the overlay window in the operation area of ​​the user interface; wherein the overlay window is used to display interactive animations, the interactive animations being associated with the operation performed by the agent on the target application associated with the interactive task.

[0217] In one implementation of this application embodiment, the interactive device 1200 may further include: The fourth processing module is used to update the state of the agent to the execution state in response to the duration indicated by the recovery strategy and the absence of the second target event, so as to control the agent to resume the execution of the interactive task.

[0218] In one implementation of this application, the first target event associated with the pause state includes at least one of the following: a size change event and / or position change event of the application window of the target application within the operation area associated with the interaction task; wherein the application window is used to display the operation information of the agent on the target application; a target occlusion event where the application window switches from an unoccluded state to an occluded state; a state switching event where the target application switches from a foreground state to a background state; an incoming call event; a newly received voice interaction event in response to the task instruction being a voice instruction; wherein the voice interaction event is used to pause the execution of the interaction task; a pause event sent from the cloud; a state switching event where the voice interaction function switches from an available state to an unavailable state; a state switching event where the screen state switches from a bright screen state to a locked screen state; a state switching event where the screen state switches from a bright screen state to a screen-off state; a pause event initiated on the interactive controls in the state area of ​​the user interface; a screen touch event; a touch event initiated on the overlay window in the operation area; wherein the overlay window is used to display interactive animations, and the interactive animations are associated with the operations performed by the agent on the target application associated with the interaction task.

[0219] In one implementation of this application, the interactive device 1200 is applied to a vehicle, and the target occlusion event does not include the occlusion event of the turn signal pop-up.

[0220] In one implementation of this application, the interactive device 1200 is applied to a vehicle, the user interface is projected onto the vehicle by the terminal, and the first target event associated with the paused state also includes: a pop-up reminder event when the terminal first projects its screen onto the vehicle.

[0221] In one implementation of this application, the first target event or the second target event associated with the stopped state includes at least one of the following: a stop event sent from the cloud; a newly received voice-based user interaction event in response to the task instruction being a voice instruction; wherein the voice-based user interaction event is used to stop the execution of the interaction task and / or execute a new interaction task; a stop event initiated on the interactive controls in the status area of ​​the user interface; wherein the status area is used to display prompt information matching the state of the agent; and a newly received non-voice-based user interaction event in response to the task instruction being a voice instruction.

[0222] In one implementation of this application, the interactive device 1200 is applied to a vehicle, the user interface is a terminal projected onto the vehicle, and the first target event or second target event associated with the stationary state also includes: a connection disconnection event between the terminal and the vehicle.

[0223] In one implementation of this application, the second target event associated with the execution state includes at least one of the following: a call termination event; a continuation event initiated by an interactive control in the state area of ​​the user interface; wherein the state area is used to display prompt information matching the state of the agent; a click event on a target option among multiple candidate options associated with a question sent by the agent in the operation area of ​​the user interface; wherein the operation area is used to instruct the agent on operation information and / or interactive animations of the target application associated with the interactive task; a recovery event in which the application window of the target application in the operation area is restored from an obscured state to an unobscured state; wherein the application window is used to display operation information; a state switching event in which the target application switches from a background state to a foreground state; a state switching event in which the screen state switches from a locked state to a lit state; a state switching event in which the screen state switches from a sleep state to a lit state; and a state switching event in which the voice interaction function is restored from an unavailable state to an available state.

[0224] In one implementation of this application embodiment, the interactive device 1200 may further include: The fifth processing module is configured to perform at least one of the following: in response to the completion of the interaction task by the agent, update the state of the agent to the completed state; in response to the completion of the interaction task by the agent, hide the state area in the user interface; wherein the state area is used to display prompt information matching the state of the agent; in response to the completion of the interaction task by the agent, hide the overlay window in the operation area of ​​the user interface; wherein the overlay window is used to display interactive animations, and the interactive animations are associated with the operation performed by the agent on the target application associated with the interaction task.

[0225] In one implementation of this application embodiment, the interactive device 1200 may further include: The clearing module is used to clear the agent's executed operations and cached data in response to the agent's state being updated to a stopped or completed state.

[0226] It should be noted that the foregoing explanation of any embodiment of the interaction method also applies to the interaction device of that embodiment, and will not be repeated here.

[0227] In the interactive device of this application embodiment, by introducing a conflict management mechanism for the agent's state, events are monitored or received in real time during the execution of interactive tasks. When a first target event that conflicts with the agent's current actual state is received, the agent's state is dynamically updated to a first target state that matches the first target event. This achieves closed-loop adaptive adjustment of task execution, effectively avoiding misoperation or task failure caused by state mismatch, thereby significantly improving the robustness, security, task success rate, and user experience of the agent in real human-machine coexistence scenarios.

[0228] To implement the above embodiments, this application also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the interaction method as described in any of the foregoing embodiments.

[0229] Figure 13 This is a schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this application. For example, the electronic device 1300 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0230] Reference Figure 13 The electronic device 1300 may include one or more of the following components: a processing component 1302, a memory 1304, a power component 1306, a multimedia component 1308, an audio component 1310, an input / output (I / O) interface 1312, a sensor component 1314, and a communication component 1316.

[0231] Processing component 1302 typically controls the overall operation of electronic device 1300, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 1302 may include one or more processors 1320 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 1302 may include one or more modules to facilitate interaction between processing component 1302 and other components. For example, processing component 1302 may include a multimedia module to facilitate interaction between multimedia component 1308 and processing component 1302.

[0232] Memory 1304 is configured to store various types of data to support the operation of electronic device 1300. Examples of this data include instructions for any application or method operating on electronic device 1300, contact data, phonebook data, messages, pictures, videos, etc. Memory 1304 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0233] Power component 1306 provides power to various components of electronic device 1300. Power component 1306 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 1300.

[0234] Multimedia component 1308 includes a screen that provides an output interface between the electronic device 1300 and the user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen may be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1308 includes a front-facing camera and / or a rear-facing camera. When the electronic device 1300 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0235] Audio component 1310 is configured to output and / or input audio signals. For example, audio component 1310 includes a microphone (MIC) configured to receive external audio signals when electronic device 1300 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1304 or transmitted via communication component 1316. In some embodiments, audio component 1310 also includes a speaker for outputting audio signals.

[0236] I / O interface 1312 provides an interface between processing component 1302 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0237] Sensor assembly 1314 includes one or more sensors for providing state assessments of various aspects of electronic device 1300. For example, sensor assembly 1314 may detect the on / off state of electronic device 1300, the relative positioning of components such as the display and keypad of electronic device 1300, changes in the position of a component of electronic device 1300, the presence or absence of user contact with electronic device 1300, the orientation or acceleration / deceleration of electronic device 1300, and temperature changes of electronic device 1300. Sensor assembly 1314 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1314 may also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, sensor assembly 1314 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0238] Communication component 1316 is configured to facilitate wired or wireless communication between electronic device 1300 and other devices. Electronic device 1300 can access wireless networks based on communication standards, such as WiFi, 4G, or 5G, or combinations thereof. In one exemplary embodiment, communication component 1316 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 1316 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra-Wideband (UWB), Bluetooth, and other technologies.

[0239] In an exemplary embodiment, the electronic device 1300 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0240] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1304 including instructions, which can be executed by a processor 1320 of an electronic device 1300 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0241] To implement the above embodiments, this application also proposes a vehicle, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the interaction method as described in any of the foregoing embodiments.

[0242] Figure 14 This is a block diagram illustrating a vehicle 1400 according to an exemplary embodiment. For example, vehicle 1400 may be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or other types of vehicle. Vehicle 1400 may be an autonomous vehicle, a semi-autonomous vehicle, or a non-autonomous vehicle.

[0243] Reference Figure 14 The vehicle 1400 may include various subsystems, such as an infotainment system 1410, a perception system 1420, a decision control system 1430, a drive system 1440, and a computing platform 1450. The vehicle 1400 may also include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and each component of the vehicle 1400 can be interconnected via wired or wireless means.

[0244] In some embodiments, the infotainment system 1410 may include a communication system, an entertainment system, and a navigation system, etc.

[0245] The perception system 1420 may include several sensors for sensing information about the environment surrounding the vehicle 1400. For example, the perception system 1420 may include a global positioning system (which may be a GPS system, a BeiDou system, or another positioning system), an inertial measurement unit (IMU), a lidar, a millimeter-wave radar, an ultrasonic radar, and a camera device.

[0246] The decision control system 1430 may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

[0247] The drive system 1440 may include components that provide powered motion to the vehicle 1400. In one embodiment, the drive system 1440 may include an engine, an energy source, a transmission system, and wheels. The engine may be one or a combination of internal combustion engines, electric motors, and compressed air engines. The engine is capable of converting energy provided by the energy source into mechanical energy.

[0248] Some or all of the functions of the vehicle 1400 are controlled by a computing platform 1450. The computing platform 1450 may include at least one processor 1451 and a memory 1452, the processor 1451 being able to execute instructions 1453 stored in the memory 1452.

[0249] Processor 1451 can be any conventional processor, such as a commercially available CPU. Processors may also include graphics processing units (GPUs), field-programmable gate arrays (FPGAs), systems-on-chips (SoCs), application-specific integrated circuits (ASICs), or combinations thereof.

[0250] The memory 1452 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0251] In addition to instruction 1453, memory 1452 can also store data, such as road maps, route information, vehicle position, direction, speed, and other data. The data stored in memory 1452 can be used by computing platform 1450.

[0252] In this embodiment of the application, processor 1451 may execute instruction 1453 to complete all or part of the steps of any of the above method embodiments.

[0253] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the interactive method as described in any of the foregoing method embodiments.

[0254] To implement the above embodiments, this application also proposes a computer program product having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the interaction method as described in any of the foregoing method embodiments.

[0255] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0256] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0257] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0258] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and compact disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0259] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0260] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments.

[0261] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0262] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. An interaction method, characterized in that, include: In response to receiving a task instruction, the agent is invoked to execute an interactive task matching the task instruction, based on the task instruction and the screen content of the user interface. During the execution of the interactive task, in response to receiving a first target event that conflicts with the actual state of the agent, a first target state that matches the first target event is determined; Update the state of the agent to the first target state.

2. The method according to claim 1, characterized in that, The user interface includes an operation area and a status area. During the execution of the interactive task, the method further includes at least one of the following: The operation area displays the operation information of the intelligent agent on the target application associated with the interactive task; The operation area displays interactive animations, which are associated with the operations performed by the agent on the target application. In the state area, prompts matching the state of the agent are displayed.

3. The method according to claim 2, characterized in that, The provision in the state area that displays prompts matching the state of the agent includes: The content indicated by the prompt message is determined based on the display constraints corresponding to the status area; wherein, the display constraints include at least one of the following: the spatial size parameters of the status area in the user interface, the device movement status, and the historical interaction behavior associated with the status area; The status area displays the content indicated by the prompt message.

4. The method according to claim 3, characterized in that, The step of determining the content indicated by the prompt message based on the display constraints corresponding to the status area includes at least one of the following: In response to the display constraints including the spatial size parameter, the content indicated by the prompt message is determined based on the size relationship between the spatial size parameter and the corresponding size threshold; In response to the display constraints including the device movement state, the content indicated by the prompt message is determined based on the relationship between the movement speed in the device movement state and the corresponding speed threshold. In response to the display constraints including the historical interaction behavior, the interaction activity of the status area is determined based on the historical interaction behavior, and the content indicated by the prompt information is determined based on the relationship between the interaction activity and the corresponding activity threshold.

5. The method according to claim 4, characterized in that, Determining the content indicated by the prompt message includes: In response to any of the following conditions, it is determined that the content indicated by the prompt message includes the state of the agent: The spatial dimension parameter is less than the corresponding dimension threshold; The movement speed is greater than the speed threshold; The interaction activity level is less than the corresponding activity threshold.

6. The method according to claim 4, characterized in that, Determining the content indicated by the prompt message includes: In response to any of the following, the content indicated by the prompt message is determined to include the state of the agent and the subtask description information at the current time step in the execution path associated with the interaction task: The spatial dimension parameter is greater than or equal to the corresponding dimension threshold; The movement speed is less than or equal to the speed threshold; The interaction activity level is greater than or equal to the corresponding activity threshold; The execution path is generated by the intelligent agent based on the task instructions and the screen content.

7. The method according to claim 2, characterized in that, The operation area includes the application window of the target application; The operation area displays the agent's operation information on the target application associated with the interactive task, including: The application window displays the operation information of the intelligent agent when performing the target operation on the target application; The target operation is the operation associated with the subtask description information at the current time step in the execution path associated with the interactive task; The execution path is generated by the intelligent agent based on the task instructions and the screen content.

8. The method according to claim 7, characterized in that, The operation area also includes a mask window that surrounds the application window; The interactive animation effects displayed in the operation area include: The overlay window displays interactive animations when the agent performs the target operation on the target application.

9. The method according to claim 1, characterized in that, The step of responding to receiving a first target event that conflicts with the actual state of the agent and determining a first target state that matches the first target event includes: In response to receiving a first target event that conflicts with the actual state, determine the conflict detection type to which the first target event belongs; From the state machine of the agent, a first target state that matches the conflict detection type is determined.

10. The method according to claim 1, characterized in that, In response to the first target state being a paused state for controlling the agent to suspend the execution of the interactive task, the method further includes: Determine a recovery strategy associated with the operational source of the first target event; wherein the recovery strategy is used to indicate the allowed duration and / or countdown duration of the paused state; In response to the fact that the duration of the paused state of the agent does not reach the duration indicated by the recovery strategy, and a second target event that conflicts with the first target state is received, a second target state that matches the second target event is determined from the state machine of the agent; Update the state of the agent to the second target state.

11. The method according to claim 10, characterized in that, In response to the second target state being an execution state for controlling the agent to resume executing the interactive task, the method further includes at least one of the following: In the operation area of ​​the user interface, the operation data of the intelligent agent on the target application continues to be displayed; In the operation area of ​​the user interface, interactive animations continue to be displayed, wherein the interactive animations are associated with the operations performed by the agent on the target application; In the status area of ​​the user interface, prompts matching the execution state are displayed.

12. The method according to claim 10, characterized in that, In response to the first target state or the second target state being a stopped state for controlling the agent to stop executing the interactive task, the method further includes at least one of the following: Hide the status area in the user interface; wherein the status area is used to display prompt information that matches the status of the agent; Hide the overlay window in the operation area of ​​the user interface; wherein the overlay window is used to display interactive animations, and the interactive animations are associated with the operations performed by the agent on the target application associated with the interactive task.

13. The method according to claim 10, characterized in that, The method further includes: In response to the duration reaching the duration indicated by the recovery strategy and no second target event being received, the state of the agent is updated to the execution state to control the agent to resume execution of the interactive task.

14. The method according to claim 1, characterized in that, The first target event associated with the paused state includes at least one of the following: The size change event and / or position change event of the application window of the target application associated with the interactive task; wherein, the application window is used to display the operation information of the agent on the target application; The target occlusion event where the application window switches from an unoccluded state to an occluded state; The state switching event where the target application switches from foreground to background state; Incoming call incident; In response to the task instruction being a voice instruction, a new received voice interaction event is added; wherein, the voice interaction event is used to pause the execution of the interaction task; Pause events sent from the cloud; The state switching event where the voice interaction function changes from an available state to an unavailable state; The screen state transition event that changes from on screen to locked screen; The screen state transition event that changes from on-screen to off-screen state; A pause event initiated on the interactive controls in the state area; Screen touch events; A touch event initiated by the masked window in the operation area of ​​the user interface; wherein the masked window is used to display the interactive animation, and the interactive animation is associated with the operation performed by the agent on the target application associated with the interactive task.

15. The method according to claim 10, characterized in that, The first target event or the second target event associated with the stopped state includes at least one of the following: Stop event sent from the cloud; In response to the task instruction being a voice instruction, a new user interaction event in the form of voice is received; wherein, the user interaction event in the form of voice is used to stop the execution of the interaction task and / or execute a new interaction task; A stop event initiated on the interactive controls in the state area of ​​the user interface; wherein, the state area is used to display prompt information matching the state of the agent; In response to the task instruction being a voice instruction, additional non-voice user interaction events are received.

16. The method according to claim 15, characterized in that, The method is performed by the vehicle, and the user interface is projected from the terminal onto the vehicle. The first target event or the second target event associated with the stopped state further includes: a connection disconnection event between the terminal and the vehicle.

17. The method according to claim 10, characterized in that, The second target event associated with the execution state includes at least one of the following: Call termination event; A continue event initiated on the interactive controls in the state area of ​​the user interface; wherein, the state area is used to display prompt information matching the state of the agent; A click event on a target option among multiple candidate options associated with a rhetorical question sent by the agent in the operation area of ​​the user interface; wherein, the operation area is used to indicate the operation information and / or interactive animations of the agent on the target application associated with the interactive task; The event that restores the application window of the target application within the operation area from an obscured state to an unobscured state; wherein, the application window is used to display the operation information; The state transition event where the target application switches from background to foreground state; The screen state transition event that changes from locked screen to on screen; The screen state transition event that changes from screen-off state to screen-on state; The state transition event where the voice interaction function is restored from an unavailable state to an available state.

18. The method according to any one of claims 1-17, characterized in that, The method further includes at least one of the following: In response to the completion of the interactive task by the agent, the state of the agent is updated to the completed state; In response to the completion of the interactive task by the intelligent agent, the state area in the user interface is hidden; wherein, the state area is used to display prompt information matching the state of the intelligent agent; In response to the completion of the interactive task by the intelligent agent, the overlay window in the operation area of ​​the user interface is hidden; wherein, the overlay window is used to display interactive animations, and the interactive animations are associated with the operation performed by the intelligent agent on the target application associated with the interactive task.

19. The method according to any one of claims 1-17, characterized in that, The method further includes: In response to the state of the agent being updated to a stopped state or a completed state, the agent's executed operations and cached data are cleared.

20. An interactive device, characterized in that, include: An execution module is used to respond to a received task instruction by invoking an agent to perform an interactive task matching the task instruction based on the task instruction and the screen content of the user interface. The determination module is used to determine a first target state that matches the first target event in response to receiving a first target event that conflicts with the actual state of the agent during the execution of the interactive task. The update module is used to update the state of the agent to the first target state.

21. The apparatus according to claim 20, characterized in that, The user interface includes an operation area and a status area, and the device further includes: A display module is configured to synchronously display at least one of the following during the execution of the interactive task: The operation area displays the operation information of the intelligent agent on the target application associated with the interactive task; The operation area displays interactive animations, which are associated with the operations performed by the agent on the target application. In the state area, prompts matching the state of the agent are displayed.

22. The apparatus according to claim 21, characterized in that, The display module is specifically used for: The content indicated by the prompt message is determined based on the display constraints corresponding to the status area; wherein, the display constraints include at least one of the following: the spatial size parameters of the status area in the user interface, the device movement status, and the historical interaction behavior associated with the status area; The status area displays the content indicated by the prompt message.

23. The apparatus according to claim 22, characterized in that, The display module is specifically used to perform at least one of the following: In response to the display constraints including the spatial size parameter, the content indicated by the prompt message is determined based on the size relationship between the spatial size parameter and the corresponding size threshold; In response to the display constraints including the device movement state, the content indicated by the prompt message is determined based on the relationship between the movement speed in the device movement state and the corresponding speed threshold. In response to the display constraints including the historical interaction behavior, the interaction activity of the status area is determined based on the historical interaction behavior, and the content indicated by the prompt information is determined based on the relationship between the interaction activity and the corresponding activity threshold.

24. The apparatus according to claim 20, characterized in that, In response to the first target state being a paused state for controlling the agent to suspend the execution of the interactive task, the device further includes: A first processing module is configured to determine a recovery strategy associated with the operation source of the first target event; wherein the recovery strategy is configured to indicate the allowed duration and / or countdown duration of the paused state; in response to the agent being in the paused state for a duration not reaching the duration indicated by the recovery strategy, and receiving a second target event that conflicts with the first target state, a second target state matching the second target event is determined from the agent's state machine; and the state of the agent is updated to the second target state.

25. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 19.

26. A non-transitory computer-readable storage medium having computer program instructions stored thereon, characterized in that, When executed by a processor, the program instructions implement the steps of the method described in any one of claims 1 to 19.

27. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 19.

Images