An interaction method, device, medium and equipment of a wearable device

Abstract

The embodiment of the specification discloses an interaction method of a wearable device, the wearable device applied by the method comprises an image sensor and an audio sensor, each environment memory unit is obtained at a fixed frequency by analyzing and processing video data continuously collected by the image sensor. Then, in response to the interaction request of the user, a serialized to-be-executed subtask and an input resource label required for the execution of the subtask can be planned. According to the execution order, at least one of historical data in the environment memory unit and current sensor-collected perception data is scheduled according to the input resource label for each subtask in sequence to execute the subtask. Finally, natural language description response information is generated based on the execution result after the execution of each subtask is completed. By constructing the time-sequenced environment memory unit, the problem caused by the lack of continuous perception and memory capability is solved, the multi-modal information collaborative reasoning is supported by the label-driven resource scheduling, and the flexible execution of the task is realized.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to an interaction method, apparatus, storage medium and device for a wearable device. Background Technology

[0002] In recent years, with the rapid development of technologies such as Augmented Reality (AR) and Virtual Reality (VR), wearable devices have become increasingly popular, especially head-mounted wearable devices such as smart glasses, which have attracted much attention due to their portability and immersive experience. Furthermore, these devices generally support voice interaction and, thanks to their head-mounted design, can easily acquire images and videos from the user's first-person perspective, further increasing the ways to interact. Therefore, how to better facilitate interaction based on wearable devices has become a key focus.

[0003] Taking smart glasses as an example, in existing technologies, the interaction process generally involves the device responding to a user's voice command by collecting at least one of the following: a video of a preset duration, an audio recording of a preset duration, or an image. Then, models are used to process the data from different modalities. When multimodal data exists, feature concatenation is typically performed. Finally, a decision module matches the concatenated features to preset rules or scripts and executes the corresponding operation. For example, generating a voice or text response, or triggering the device to perform a task. Audio data can be converted into text commands, while image data or video frames are used to identify objects, scenes, or classify actions.

[0004] However, the current technology's method of triggering and then acquiring data for interaction lacks the ability to perceive the user's surrounding environment and does not possess contextual memory. Because it executes an operation only once, the device struggles to interact across time. When faced with long-term tasks requiring multi-step collaboration, the interaction process exhibits severe discontinuities. Users cannot engage in continuous dialogue based on historical states, and the device cannot proactively provide coherent task guidance. Furthermore, multimodal fusion and fixed task scripts cannot handle non-standard instructions and undefined situations, remaining at a low level of one-to-one interaction.

[0005] Based on this, this specification provides an interaction method for wearable devices to partially solve the problems existing in the prior art. Summary of the Invention

[0006] This specification provides an interaction method, apparatus, storage medium, and electronic device for wearable devices to partially solve the problems existing in the prior art.

[0007] The embodiments in this specification adopt the following technical solutions: This specification provides an interaction method for a wearable device, wherein the wearable device's sensors include at least an image sensor and an audio sensor, and the method includes: The video data continuously acquired by the image sensor is analyzed and processed according to a preset frequency to obtain several environmental memory units divided according to time sequence. In response to user interaction requests, a serialized subtask to be executed is planned and generated, along with the input resource tags required for the execution of the subtask; According to the execution sequence of each subtask, for each subtask, the required input resource tag scheduling data is based on the subtask execution, and the scheduling data includes historical data retrieved from the environmental memory unit and / or sensing data collected from the sensors of the wearable device; Execute the subtask based on the data scheduled for it. When each subtask is completed, a response message in natural language is generated and returned based on the execution results of each subtask and the interaction request.

[0008] This specification provides an interactive device for a wearable device, wherein the wearable device's sensors include at least an image sensor and an audio sensor, and the device includes: The environment memory module is used to analyze and process the video data continuously acquired by the image sensor according to a preset frequency to obtain a number of environment memory units divided according to time sequence. The task planning module is used to respond to user interaction requests, plan and generate serialized subtasks to be executed, and the input resource tags required for the execution of the subtasks; The demand determination module is used to schedule the input resource tag data required for the execution of each subtask according to the execution sequence of each subtask. The scheduled data includes historical data retrieved from the environment memory unit and / or sensing data collected from the sensors of the wearable device. The task execution module is used to execute the subtask based on the data scheduled for it. The interaction module is used to generate and return response information in natural language based on the execution results of each subtask and the interaction request when each subtask is completed.

[0009] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the interaction method of the wearable device described above.

[0010] This specification provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the interaction method of the wearable device described above.

[0011] The above-described at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects: This specification discloses an interaction method for a wearable device. The wearable device includes an image sensor and an audio sensor. By analyzing and processing continuously acquired video data from the image sensor, environmental memory units are obtained at a fixed frequency. In response to user interaction requests, a sequence of subtasks to be executed, along with input resource tags required for subtask execution, can be planned. For each subtask, execution is performed sequentially according to the execution order, scheduling at least one of historical data from the environmental memory units and currently acquired sensor data based on the input resource tags. Finally, after each subtask is completed, a response information describing natural language is generated based on the execution results. By constructing a temporally sequenced environmental memory unit, the problem caused by a lack of continuous perception and memory capabilities is solved. Through tag-driven resource scheduling, collaborative reasoning of multimodal information is supported, enabling flexible task execution. Attached Figure Description

[0012] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings: Figure 1 An interactive flowchart of a wearable device provided in an embodiment of this specification; Figure 2 An architecture diagram of multi-agent interaction provided in the embodiments of this specification; Figure 3 This is a schematic diagram of the training process provided in the embodiments of this specification; Figure 4 A schematic diagram of an interactive device for a wearable device provided in an embodiment of this specification; Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this specification. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0014] Voice interaction solutions for wearable devices are typically designed around dialogue, relying on clear command and response loops when handling tasks such as software development. However, these solutions struggle to adapt to continuous multimodal data inputs such as video streams, and their interaction mechanisms cannot support the continuous parsing and flexible transfer of real-time video data.

[0015] This specification proposes an interaction method for wearable devices, enabling persistent perception of continuous environmental information and context-based dynamic planning and execution of multimodal tasks. The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0016] Figure 1 A flowchart illustrating the interaction of a wearable device provided in this specification includes the following steps: S100: The video data continuously acquired by the image sensor is analyzed and processed according to a preset frequency to obtain a number of environmental memory units divided according to time sequence.

[0017] In the embodiments described in this specification, the following are employed: Figure 1 The device through which the wearable device interacts can be any electronic device, such as a head-mounted device, a computer, a server, or a server cluster consisting of multiple servers. If the interaction process is performed by a non-local device, the wearable device needs to send the collected data to the corresponding device for execution, such as sending it to a server. For ease of description, the following explanation will only use the example of a server executing the interaction process.

[0018] To address the issues of information gaps in interaction caused by the lack of continuous environmental awareness and contextual memory in wearable devices such as smart glasses, and the difficulty in handling complex, long-chain tasks due to insufficient multimodal information fusion and the absence of dynamic programming mechanisms, this specification provides an interaction method for wearable devices based on multi-agent collaboration. By introducing a continuous video stream analysis process, it achieves the time-series recording and structured storage of the user's historical environmental information. This enables the server to maintain continuous contextual awareness during interaction, solving the information gap problem caused by traditional trigger-based interaction. Based on coherent environmental understanding, the server can accurately decompose complex user requests into sequential tasks and dynamically fuse historical memory and real-time perception data to execute each step through a resource-aware scheduling mechanism. This generates coherent natural language interaction responses, significantly improving assisted intelligence and interaction fluency in real-world scenarios.

[0019] Specifically, the wearable device's sensors include at least an image sensor and an audio sensor. As mentioned earlier, due to the characteristics and convenience of voice interaction, wearable devices often include voice-based interaction methods; therefore, an audio sensor can be included in the wearable device. This audio sensor is an important data source for recognizing user intent; for example, by using voice recognition functions to determine the user's intent. The image sensor can be used to collect video data and is a data source for constructing data on the user's surrounding environment, complementing the audio data.

[0020] Video data acquired by image sensors can be considered equivalent to data acquired through human vision. It serves as the source data for servers to understand the user's location, identify objects, and analyze scene conditions. Because continuous environmental awareness is required to maintain consistent contextual understanding, image sensors need to continuously acquire video data.

[0021] Of course, the embodiments in this specification do not limit the types of sensors in the wearable device to these two categories. Other sensors, such as lidar, depth sensors, inertial measurement units, and biosensors, can be used as needed. Furthermore, even if only image sensors and audio sensors are used, the embodiments in this specification do not limit the number of sensors. For example, multiple image sensors can be used to achieve a wider field of view, or multiple audio sensors can be used to achieve better noise reduction and sound source localization.

[0022] In the embodiments described in this specification, the server can continuously monitor and process video data collected by the image sensor of the wearable device, analyze and process it according to a preset frequency, and obtain a number of environmental memory units divided according to time sequence.

[0023] This is because, in order to achieve a continuous understanding of the user's environment and the coherence of historical context, the interaction method in the embodiments of this specification first needs to construct a temporally sequenced environmental memory bank. This memory bank records the environmental changes and event developments experienced by the user through the wearable device over time, serving as the foundation for subsequent task planning and execution. Traditional interaction methods only analyze the screen at the moment the command is triggered in isolation, failing to connect it to previous or subsequent environmental states, leading to breaks in the interaction process. By sampling, analyzing, and storing environmental information at a fixed frequency, the server can construct a dynamically growing environmental memory sequence over time, allowing subsequent steps to utilize past history.

[0024] First, the server can acquire video data sent by the image sensor of the wearable device. This video data can be streamed, or the wearable device can send video data collected by its own image sensor to the server at preset intervals. The server then analyzes and processes the video data to determine the environmental memory unit. If the latter method is used, the preset intervals followed by the wearable device correspond to the aforementioned preset frequency.

[0025] The preset frequency can be configured based on the timeliness requirements of the actual scenario and the available computing resources, such as sampling once per second or once every five seconds, thereby extracting a series of video data segments representing different moments in the video stream. Through this equal-interval or adaptive-interval sampling, the server transforms the continuous video stream into a discrete but coherent set of video data segments.

[0026] The server can then analyze and process the acquired video data segments to determine the scene semantic information and scene summary of each video data segment.

[0027] The server can extract structured scene semantic information from video data segments using a pre-defined visual analysis model or scene understanding model. This scene semantic information may include a list of identified objects, the types of detected activities, the overall classification of the scene, and the spatial relationships between objects, etc.

[0028] Simultaneously, the server can generate a text description as a scene summary for the segment based on the extracted semantic information. For example, for a video data segment from a cooking instruction scene, the analysis and processing may determine that its scene semantic information includes "cutting board," "tomato," "knife," and "cutting action," and generate a scene summary of "the user is cutting tomatoes" accordingly.

[0029] Then, the timestamps, scene semantic information, and scene summaries of the video data segments are encapsulated to determine the environment memory units.

[0030] After analysis, the video data segment contains other information, which the server can integrate and encapsulate. At a minimum, the acquisition timestamp, the analyzed scene semantic information, and the generated scene summary can be encapsulated to form a complete data structure—an environmental memory unit. This environmental memory unit, as the smallest unit for recording the instantaneous state of the environment, provides its precise location on the timeline with its timestamp, structured data for machine processing with its scene semantic information, and a scene summary for quick retrieval during subsequent calls and for human reading.

[0031] Finally, the environmental memory unit is stored and determined according to the timestamp.

[0032] The server can store environment memory units in an ordered database or cache structure according to the chronological order of their timestamps. This timestamp-ordered storage method logically forms a time-series memory, i.e., an environment memory bank. When it is necessary to understand the current state or rewind history, the server can efficiently query and retrieve data from this memory bank based on time range or semantic content, thereby providing historical context when necessary. Through this process, the server realizes the transformation from raw video streams to time-series environment memory, providing convenient conditions for subsequent business operations to utilize historical data.

[0033] Alternatively, in the embodiments described in this specification, this process can also be handled by an environmental memory agent configured within the server. This agent continuously runs in a background silent mode, sampling the video stream continuously received from the image sensor of the wearable device at a preset frequency, thereby acquiring a series of discrete but coherent video data segments. Unlike the server passively executing instructions, the environmental memory agent independently and actively maintains the rhythm of information acquisition from the environment, ensuring the continuity and timeliness of memory, without waiting for user instructions to trigger it.

[0034] By introducing an environmental memory agent, continuous perception and memory can be transformed from a passive, server-invoked function into an autonomous behavior, enabling context accumulation and updates that are independent of user requests and have low business logic complexity. This environmental memory agent, as a persistent cognitive module, ensures the uninterrupted accumulation of environmental context, thereby resolving the state fragmentation problem caused by fragmented information processing in traditional interactions.

[0035] Due to the aforementioned advantages of intelligent agents in this interaction process, subsequent steps can be executed by pre-defined intelligent agents, or fundamentally, by the server. From a feasibility standpoint, whether implemented by the server or by intelligent agents set up within the server, there are no technical barriers. Therefore, the following explanations will use intelligent agents executing each step as an example, but do not limit implementation to intelligent agents.

[0036] S102: In response to the user's interaction request, plan and generate serialized subtasks to be executed, as well as the input resource tags required for the execution of the subtasks.

[0037] In the embodiments described in this specification, in response to a user's interactive request, the server can transform it into an execution plan containing resource guidance. User interactive requests expressed in natural language often contain a complex goal requiring multiple actions to be coordinated. Traditional processing methods either attempt to execute this as a single command, resulting in an inoperable outcome due to the overly general goal. Alternatively, using fixed response templates limits responses to formulaic commands, failing to adapt to the flexible and personalized requests in real-world scenarios. Therefore, by breaking down high-level interactive requests into a series of simple and executable sub-tasks, and pre-defining the data resources required for each sub-step, the user's seemingly simple but precise instructions can be transformed into multiple interconnected sub-tasks. In subsequent execution phases, the server can provide accurate and clear goals and explicit resource inputs for each sub-task, enabling the execution of complex and ambiguous tasks.

[0038] Specifically, after receiving an interaction request from a user uploaded via a wearable device, the server can invoke a pre-defined scheduling agent, which will then determine how to handle the interaction request. This scheduling agent can invoke a task planning agent to convert the interaction request into text and send it to the task planning agent.

[0039] The task planning agent, based on the interaction requests sent by the scheduling agent, determines the task and breaks it down into subtasks. First, the task planning agent performs semantic understanding of the interaction requests to identify the target state that the user hopes to achieve. Then, based on common sense or knowledge of the task domain, it reverse-engineers the target state to obtain a series of logically connected subtasks to be executed.

[0040] For example, when a user sends an interactive request to "write a creative summary of the painting I just finished and save it", the task planning module may plan it as three sequential sub-tasks to be executed: "identify and analyze the content of the painting in the current field of view", "generate a text summary based on the analysis results", and "store the text summary to a specified location".

[0041] Furthermore, while generating a sequence of subtasks, the task planning agent can also analyze and label the input resources required for the execution of each subtask. These labels are key metadata for executing the subtask, and the input resource labels are used to indicate the data types or data sources that the agent depends on for executing the corresponding subtask.

[0042] Based on this input resource label, the agent can determine whether performing a subtask requires relying on the latest perception of the current environment, obtaining context from historical records, or both. For example, for the subtask of "identifying and analyzing the content of a painting in the current field of view," the input resource label would be marked as requiring "real-time visual perception data"; while for the subtask of "generating a text summary," the label might be marked as requiring "historical environment memory units" to obtain the analysis results from previous steps. These labels together constitute a list of resource requirements.

[0043] Finally, the task planning agent serializes the subtasks to be executed and their corresponding input resource tags, packages them into a structured task execution plan, and returns it to the scheduling agent. Through this step, the scheduling agent and the task planning agent in the server work together to transform a vague natural language command from the user into a series of subtasks that a machine can execute, thus providing the necessary information for subsequent on-demand scheduling of data and execution actions.

[0044] It should be noted that, to ensure the subtask can be executed, the subtask generated by the task planning agent contains the execution logic of that subtask, and this execution logic is structured data. Therefore, this execution logic actually contains at least the specific operation definition of the subtask. This execution logic can be designed as a data object that encapsulates complete operation instructions and their execution constraints, typically but not limited to using machine-readable serialization formats such as JSON and Protocol Buffers.

[0045] For example, the subtask may contain a globally unique identifier to uniquely identify it within the task sequence. The operation type, such as retrieve, analyze, compare, generate, or activate, refers to the specific parameters or context required to perform the operation. If the operation is retrieve and the resource tag is historical data (requires history), the parameters might include the time window, target object, and event type. It also includes the aforementioned input resource tag, and may contain the data format output to the next subtask after the subtask is completed—that is, the output format.

[0046] S104: According to the execution sequence of each subtask, for each subtask, the required input resource tag scheduling data is used to execute the subtask. The scheduling data includes historical data retrieved from the environmental memory unit and / or sensing data collected from the sensors of the wearable device.

[0047] In the embodiments of this specification, in order to execute each subtask based on the required data and achieve efficient on-demand utilization of computing and data resources, a dynamic data scheduling mechanism is implemented based on input resource tags during the task execution phase. For each subtask, the task planning agent labels the input resource tags required by the subtask according to its needs. These input resource tags indicate the nature of the data on which the subtask execution depends, whether it depends on historical context from past records or on real-time perception of the current environment. If all possible data sources are fetched simultaneously for all subtasks without distinction, it will result in redundant data transmission and processing overhead and may introduce interference from irrelevant information. Therefore, scheduling corresponding historical data or real-time perceived data according to the specific needs of the subtask can improve processing efficiency and reduce resource consumption.

[0048] Specifically, the scheduling agent in the server processes each subtask to be executed sequentially according to the task execution sequence. For the subtask that needs to be processed, the scheduling agent first parses its attached input resource tag and takes different data scheduling paths based on the indication of the tag.

[0049] When the parsing determines that the input resource label of the subtask is a historical data label, it indicates that the execution of the subtask depends on the context information of past events or environment, and the scheduling agent can send the subtask to the preset memory retrieval agent.

[0050] After receiving a subtask, the memory retrieval agent constructs a historical data query request based on the subtask's execution logic and input resource tags. It then performs querying, matching, and sorting operations from all environmental memory units stored on the server, retrieving several environmental memory units related to the current subtask from these chronologically arranged structured records.

[0051] The scheduling agent can then receive the environmental memory unit returned by the memory retrieval agent as historical data that needs to be scheduled.

[0052] For example, for a subtask that requires "comparing the cleanliness of the room to that of ten minutes ago," the memory retrieval agent will locate and retrieve environmental memory units generated approximately ten minutes ago. These retrieved environmental memory units, as historical data for scheduling required to execute the subtask, are returned to the scheduling agent for later use.

[0053] In the process of retrieving contextual memory units, the memory retrieval agent determines to search the contextual memory unit library based on the input resource tag. It further analyzes the execution logic of this subtask to extract information such as the time range, semantic topic, or key entities required for the retrieval. Based on the extracted information, the memory retrieval agent constructs a structured query request that specifies the time and content matching conditions for the retrieval within the contextual memory units. Subsequently, the memory retrieval agent executes this data query process, retrieving several matching contextual memory units and returning them as historical data.

[0054] When the parsing determines that the input resource label for the subtask is a sensory data label, it indicates that the execution of the subtask requires obtaining the current state of the environment. The scheduling agent can then further parse the sensory data label to determine the specific type of the required sensory data. For example, is it acceleration data, image data, audio data, or user physiological data that needs to be acquired?

[0055] After determining the type of sensed data, the scheduling agent sends a command to the wearable device connected to the server, specifying which sensor's real-time data stream needs to be acquired. Upon receiving the command, the wearable device identifies the matching sensor from its integrated sensors, activates that sensor to acquire its data, or acquires real-time sensed data from the continuously operating sensor.

[0056] The collected perception data is then returned to the server, where it is retrieved by the scheduling agent. The wearable device contains at least an image sensor and an audio sensor, thus it can collect perception data through at least these two types of sensors.

[0057] Through the two differentiated scheduling paths described above, the scheduling agent can obtain corresponding data for different subtasks, thereby avoiding unnecessary data processing overhead while ensuring the accuracy of task execution, and realizing intelligent processing and refined acquisition of resource scheduling.

[0058] By adding continuous recording to the environmental memory unit and a mechanism for retrieving historical data, the problem of lacking sustained environmental memory capabilities is solved. Unlike solutions that can only react to the current instantaneous scene, this specification uses a time-sequential environmental memory unit to give smart glasses capabilities similar to human "work experience" and "scenario recall." When handling complex requests involving time dimensions or state changes, such as answering "Where did I put my keys?" or providing guidance on "Which screw should I tighten next?", the server's scheduling agent can quickly retrieve relevant historical scenarios and operation steps from the database, stringing together each independent interaction into a coherent, contextualized dialogue. This not only significantly improves support for long-term tasks and complex question-and-answer capabilities but also enables wearable devices supported by the server to learn user habits and preferences from past interactions, providing more accurate responses when interacting with users.

[0059] S106: Execute the subtask based on the data scheduled for it.

[0060] In the embodiments of this specification, after the data scheduling is completed, the scheduling agent in the server can execute each subtask sequentially according to the planned execution order of the subtasks based on the data scheduled for the subtask.

[0061] The subtasks generated in the task planning stage of step S102 have different types and objectives. For example, some subtasks require complex visual reasoning, while others only require simple data recording or device control. If a single general-purpose module handles all types of subtasks, it may lead to low execution efficiency. If a dedicated module is used to execute the corresponding subtasks, a unified coordination mechanism is required. In the embodiments of this specification, the scheduling agent plans the execution order and schedules the corresponding agent to execute each subtask.

[0062] Specifically, the scheduling agent can execute steps S104 and S106 sequentially. That is, after scheduling the data required for each subtask, it executes each subtask according to the execution order (i.e., the execution order recorded in the serialized subtasks). Alternatively, it can execute the scheduling data process corresponding to step S104 for each subtask in the execution order, and then execute step S106. Steps S104 and S106 are repeated for each subtask to gradually complete the execution of each subtask. This specification does not restrict which method is used; it can be set as needed.

[0063] For ease of description, we will use the execution process of a single subtask as an example. Whether it's the retrieved historical environmental memory unit or the real-time collected sensor data, once all the data required for the subtask to be executed is scheduled and ready, the scheduling agent fulfills its role as an execution distribution center. Based on the type attributes of the subtask determined during the planning phase, it is sent along with the scheduled data to the pre-defined specialized agent corresponding to that task type.

[0064] Among them, the corresponding type of intelligent agent refers to an intelligent agent set up in the server that is specifically designed to handle a certain type of operation, such as an intelligent agent responsible for visual question answering, an intelligent agent responsible for controlling the device, or an intelligent agent responsible for generating text summaries.

[0065] Upon receiving the data packet from the scheduling agent, the corresponding type of agent immediately initiates its internal processing logic to formally execute the subtask. It uses the input data as parameters for its algorithm, runs it, and produces the result required by the subtask. The result produced by this agent can be diverse; for example, it might be text describing the current scene, a control command sent to an external device, or a calculated judgment.

[0066] After completing its computation and operation, the agent of this type needs to return the execution result to the scheduling agent. The scheduling agent is then responsible for temporarily storing or recording this result and using it as part of the context to trigger the scheduling and execution of the next subtask in the sequence, until the entire task sequence is completed.

[0067] It should also be noted that, in the embodiments of this specification, to adapt to the ever-growing range of real-world application scenarios, the interaction process supports including external third-party intelligent agents within the scope of the scheduling intelligent agent's invocation during the execution phase. In complex tasks in real-world environments, user needs may exceed the perception and computing capabilities of the server or wearable device itself, such as operations involving flight inquiries, hotel reservations, or online payments.

[0068] If the interaction process is limited to the functions of the server or service provider itself, it will be difficult to complete such open tasks. Therefore, in the embodiments of this specification, external service capabilities can be abstracted and encapsulated as intelligent agents that can be scheduled and driven by a schedulable intelligent agent, or when a third party provides such an intelligent agent, the third-party intelligent agent can be used as an intelligent agent that can be scheduled and driven by a schedulable intelligent agent.

[0069] Specifically, the corresponding types of intelligent agents include not only those deployed internally on the server for handling core functions such as perception, memory, and dialogue, but also external third-party intelligent agents accessed through standardized interfaces, such as flight query agents, hotel booking agents, or payment processing agents. When these external intelligent agents access the server, they can register their service type, input data format, and output result specifications with the scheduling agent through the server's own scheduling protocol.

[0070] When the scheduling agent determines, based on the type of the subtask, that it needs to be executed by a registered external third-party agent, its workflow remains consistent with that of calling an internal agent. The scheduling agent encapsulates the data required to execute the subtask according to the protocol format registered by the target external agent and sends the request through a network interface. Upon receiving the request on its remote server, the corresponding third-party agent executes its business logic or service call. After execution, the third-party agent returns the execution result to the scheduling agent in the agreed-upon format. The scheduling agent receives this execution result and treats it as the execution result of the subtask, incorporating it into the subsequent task sequence progression and the final response generation process.

[0071] For example, the scheduling agent sends the user-specified check-in date and city information as the data required to execute the subtask to a third-party hotel booking agent, which can then call the inventory query interface of the partner hotels. Upon completion, the agent returns a list of bookable hotels in a pre-defined format to the scheduling agent.

[0072] S108: When each subtask is completed, a response message in natural language is generated and returned based on the execution results of each subtask and the interaction request.

[0073] To provide users with easily understandable interactive responses, and to present the execution results of the decomposed subtasks in a user-comprehensible manner, in this embodiment of the specification, after all subtasks have been executed, a dedicated natural language response generation and delivery stage returns response information to the user. The task planning and execution phase may involve the collaborative work of multiple specialized intelligent agents, and the execution results produced by each agent may be structured data, logical judgments, or operational instructions. Presenting these raw intermediate results directly to the user would result in fragmented and difficult-to-understand information. Therefore, an integration and transformation stage is needed to merge the execution results of all subtasks into a complete response to the user's interactive request, and to express this response in fluent natural language.

[0074] Specifically, after confirming that all subtasks in the task execution sequence have been completed and that the execution results returned by each subtask have been collected, the scheduling agent summarizes all relevant execution results and packages them together with the text content of the interaction request received in step S102 to form a complete response generation context package. Subsequently, the scheduling agent sends this context package to the text dialogue agent used for natural language generation.

[0075] Upon receiving the aforementioned data packet, the text-based dialogue agent performs semantic integration and language generation logic. It first parses the user's interaction request to clarify the core issue requiring a response or the operational goal needing confirmation. Next, the agent analyzes the execution results of each subtask, understanding the meaning of each result and its contribution to the overall task. Based on this, the agent logically connects, removes redundancy, and semantically refines these scattered information points, organizing and generating a text that conforms to the dialogue context—that is, a response message described in natural language.

[0076] For example, for a complex request that includes multiple sub-tasks such as "checking the weather", "planning a travel route", and "booking a taxi", the response generated by the text dialogue agent might be: "We have found that today is sunny, the smooth route from your home to the airport has been planned, and a taxi has been successfully booked to wait at the entrance of the community at 2 pm." After generating the response information, the text-based dialogue agent returns it to the scheduling agent. The scheduling agent is then responsible for sending the response information to the wearable device via the communication link between the server and the wearable device. The wearable device then presents the response information to the user through its audio output module, such as a speaker, or its visual output module, such as a projector on eyeglass lenses, completing the entire interaction process from user command input to feedback output.

[0077] based on Figure 1 The method described is an interaction method for a wearable device. The wearable device includes an image sensor and an audio sensor. By analyzing and processing continuously acquired video data from the image sensor, environmental memory units are obtained at a fixed frequency. In response to user interaction requests, a sequence of subtasks to be executed, along with input resource tags required for each subtask, can be planned. For each subtask, execution is performed sequentially according to the execution order, scheduling at least one of historical data from the environmental memory units and currently acquired sensor data based on the input resource tags. Finally, after each subtask is completed, a response information describing the result in natural language is generated. By constructing a temporally sequenced environmental memory unit, the problem caused by the lack of continuous perception and memory capabilities is solved. Through tag-driven resource scheduling, collaborative reasoning of multimodal information is supported, enabling flexible task execution.

[0078] In one or more embodiments of this specification, in order to handle different tasks through the same process and realize an interactive method that supports everything from simple question-and-answer to complex operations, user interaction requests are processed as "tasks" in a broad sense. Regardless of whether the user's interaction request is a simple information dialogue or a complex multi-step operation, the server or scheduling agent abstracts it into a "task" that needs to be planned and executed, and processes it through the same scheduling and response generation process. This allows the method to be implemented based on a concise architecture, avoiding the need to design completely separate processing channels for different types of requests.

[0079] Especially for dialogue-based requests, the processing flow is seamlessly integrated into the overall architecture. Dialogue-based requests are treated as a special task, which typically contains only one subtask focused on generating an appropriate response. This allows dialogue requests to also benefit from the resource scheduling mechanism described in the embodiments of this specification. When real-time perception or historical memory is needed to generate a response, the scheduling agent can allocate the necessary data, thereby providing context-accurate and information-rich dialogue responses, achieving a unification of technical implementation for simple interactions and complex tasks.

[0080] Specifically, when the user's interaction request is a dialogue, the generated task execution sequence is essentially a sequence containing only a single subtask. The goal of this subtask is to generate an appropriate response to the user's question or instruction. The scheduling agent, based on the input resource labels marked for this subtask, schedules the necessary data for it using the exact same mechanism as when handling complex tasks. For example, if a user asks, "What was the file I put on my desk ten minutes ago?", this subtask would be labeled as requiring "historical data," and the scheduling agent would retrieve and obtain the relevant environmental memory units from approximately ten minutes ago.

[0081] Subsequently, the scheduling agent, based on the specific nature of the dialogue subtask, sends it along with the scheduled data to the most suitable dialogue processing agent. This agent could be a text-based dialogue agent, a text-image dialogue agent, or a video dialogue agent, each specializing in handling plain text responses, responses requiring image explanations, or responses requiring video content parsing, respectively. The selected dialogue agent then uses the received data to generate a response to the dialogue subtask. This response may be structured text or may include non-text elements such as images, diagrams, or video clips.

[0082] To ultimately respond to the user via voice or other means, the response results generated by all types of dialogue agents will be sent by the scheduling agent to the text dialogue agent for integration and standardization. The text dialogue agent acts as a "response synthesizer," responsible for integrating the execution results, which may contain multimodal elements, into a natural language text description.

[0083] For example, if the video dialogue agent returns an analyzed video and several keyframes, the text dialogue agent will generate a text response such as, "Based on the analysis, your squatting motion just now had a sufficient range of motion, but please be careful not to let your knees go past your toes. Please refer to the attached image for specific posture," and will also quote the attached image. Finally, the wearable device will display this to the user.

[0084] In addition, in one or more embodiments of this specification, the server or scheduling agent may, in response to an interaction request, first determine the user's intent before determining which type of operation to perform.

[0085] User commands to wearable devices such as smart glasses vary significantly in complexity and resource requirements. Simple information queries and complex multi-step operations require drastically different processing depths and system resource support. Differentiating between these can facilitate better subsequent processing. Therefore, before planning and executing specific tasks, a dedicated intent recognition stage is introduced to determine the fundamental purpose of the user's request and allocate the appropriate processing channel accordingly.

[0086] Specifically, the scheduling agent first sends the text content of the request to the preset intent recognition agent.

[0087] This intent recognition agent has a built-in trained natural language understanding model. It performs semantic analysis and intent parsing on received interaction requests, outputting specific user intents. This involves structured parsing of the user's desired goal and the corresponding intent category. In the embodiments described in this specification, the intent categories are designed to include at least two types: "execution intents" and "dialogue intents." Execution intents refer to requests that require the system to plan and execute a series of operations to change a state or achieve a complex goal, while dialogue intents refer to requests whose primary purpose is information acquisition, knowledge answering, or simple confirmation.

[0088] For example, for the request "Send the photos I just took to Manager Wang via email", the intent recognition AI will classify it as an execution intent, while for the request "How do I set the email signature?", it will be classified as a dialogue intent.

[0089] After the intent recognition agent completes its analysis and returns results, the scheduling agent can then invoke different types of agents to handle subsequent processing based on the returned intent category. If the intent category is a dialogue intent, the scheduling agent will invoke the corresponding dialogue processing agent. For such requests, the planned "sequential subtasks to be executed" are typically simplified into a core subtask: generating an answer to the user's question. The input resource labels required for this subtask are then labeled based on whether the answer needs to incorporate real-time visual data or historical memory; for example, it might be labeled as requiring "real-time visual perception data" to identify the object the user is pointing to.

[0090] If the intent category is an execution intent, the scheduling agent will invoke a dedicated task planning agent. The task planning agent performs logical decomposition and step design based on the parsed specific user intent. See step S102 for details, which will not be elaborated upon here. Through the intent recognition and classification process, the scheduling agent can execute simple requests using a fast track, while performing in-depth planning for complex requests, thus providing a more efficient processing flow at the overall architecture level.

[0091] Furthermore, once the scheduling agent determines the user's intent through the intent recognition agent, this user intent can also be used when planning tasks and determining input resource labels.

[0092] Specifically, when the intent category is an execution intent, the scheduling agent invokes the preset task planning agent and sends the user intent to the task planning agent. The scheduling agent then sends the parsed user intent completely to the task planning agent. The user intent guides the task planning agent on how to plan the task, and is more structured and explicit than the text of the interaction request, containing the response the user hopes to receive or the result of the task to be executed.

[0093] Next, the task planning agent determines the task to be executed corresponding to the user's intent, and breaks down the task into sequential sub-tasks. Compared to the process provided in the previous embodiment, when an interaction request containing a user intent is received, the task planning agent can plan based on the user intent. First, the task planning agent analyzes the user intent to determine its corresponding task type. Although all are execution intents, different intents correspond to different business domains or problem types, with subtle differences, such as whether the user intent is "navigation," "movement guidance," "device control," or "information retrieval." Then, based on the task type, the task planning agent calls the corresponding domain's knowledge graph, action model, or planning strategy to perform task planning. This narrows the broad domain of the planning process to a specific domain, improving the accuracy and efficiency of the planning.

[0094] For example, when a user's intent is identified as "the current step to complete this dish," the task scope is defined as "cooking operation guidance." The planning agent will then automatically base its planning on knowledge of cooking-related action sequences, ingredient processing logic, and kitchen utensil usage rules, without considering irrelevant steps related to navigation or document editing, thus ensuring the rationality and professionalism of the planning results.

[0095] Finally, for each subtask, structured execution logic is generated, and the input resource labels required for the execution of that subtask are marked. This process is similar to the aforementioned step S104, and will not be described in detail here.

[0096] Similarly, in the embodiments of this specification, when the intent category is an execution intent, the scheduling agent invokes a preset dialogue agent to send the user intent to the dialogue agent.

[0097] At this point, the scheduling agent will no longer invoke the task planning agent, but instead invoke a dialogue agent with planning capabilities. This dialogue agent will plan the subtasks to be executed and determine the data required to implement the dialogue.

[0098] Through a dialogue agent, subtasks to be executed are generated based on user intent, along with structured execution logic for each subtask and labels indicating the input resources required for its execution.

[0099] Upon receiving a user's intent, the dialogue agent first analyzes the intent and generates one or more subtasks to be executed. These subtasks are not arbitrary sets of actions, but rather constitute a sequence of steps required to achieve the user's intent. Specifically, the dialogue agent constructs structured execution logic for each generated subtask and labels its input resources based on reasoning about the conditions required for successful execution of that step.

[0100] For example, given a user's intent to "summarize the whiteboard content from the meeting," the dialogue agent might generate a subtask to "identify and transcribe the text and charts on the whiteboard." It will construct execution logic for this subtask and, because it heavily relies on visual input, label it as requiring "real-time visually perceptible data" as input resources. In this way, the dialogue agent completes the process from understanding the intent to producing a structured, resource-aware preliminary execution plan, providing guidance for subsequent scheduling and execution.

[0101] Furthermore, in the examples provided in this specification, even dialogue intents may involve different types of information processing needs. Some dialogue intents require visual analysis in conjunction with real-time video streams, some require reference to static images or graphic information, while others only require pure text reasoning and responses. To improve response efficiency and resource utilization, and to provide highly adaptable interactive responses for different types of user intents, the scheduling agent can also determine which specific category of dialogue agent should perform task planning by invoking the dialogue routing agent.

[0102] Specifically, the scheduling agent first invokes the dialogue routing agent, sending the structured user intent as input for routing decisions. This dialogue routing agent has built-in routing decision logic that analyzes the received user intent to determine the appropriate dialogue channel. This dialogue channel is defined as several processing paths with different core capabilities, including at least video dialogue channels, image / text dialogue channels, and text dialogue channels.

[0103] For example, for the user intent "Describe what the flashing red indicator light on this instrument panel in front of me means," the routing decision logic will determine that understanding this intent depends on the parsing of specific visual elements in the real-time video footage, and therefore assign it to the video dialogue channel. For the user intent "Give me some dietary advice based on the nutrition facts label on the food packaging I just saw," the routing logic will determine that this intent requires text recognition and analysis in conjunction with previously captured static images, and therefore assign it to the text-to-image dialogue channel. And for the user intent "Add 'meeting at 3 PM' to my calendar," the routing logic will determine that this intent can be executed entirely through pure text semantic understanding, and therefore assign it to the text dialogue channel.

[0104] After determining the dialogue channel, the dialogue routing agent can select the agent matching the channel from among the multiple deployed dialogue-type agents as the target agent. Specifically, video dialogue channels correspond to video dialogue agents, image-text dialogue channels correspond to image-text dialogue agents, and text dialogue channels correspond to text dialogue agents.

[0105] The dialogue routing agent returns the identifier of the identified target agent to the scheduling agent. The scheduling agent can then send the user's intent to the target agent for subsequent task planning based on this identifier. This dialogue routing mechanism ensures that each user intent is guided to the matching agent, thereby achieving the effect of generating high-quality, low-latency responses.

[0106] In addition, in the embodiments of this specification, the data sources for different subtasks are different. Some subtasks need to trace the historical environmental state for comparison or continuation, while others need to perceive the latest current environment to make real-time reactions, and still others may be based entirely on internal calculations or loaded knowledge.

[0107] Therefore, labeling each subtask with input resources during the planning phase can improve task execution efficiency and reduce the probability of task failure.

[0108] Specifically, the task planning agent or dialogue agent, after generating the structured execution logic for the sub-task, performs dependency analysis on this execution logic. The dependency analysis focuses on two main points: first, determining whether successful execution of the sub-task requires retrieving relevant historical data from the environmental memory unit; and second, determining whether it requires calling the sensors of the wearable device to collect real-time perception data. Both of these judgment processes can be based on a semantic understanding of the sub-task's objective, the object of operation, and the success conditions.

[0109] For example, for a subtask whose execution logic is "comparing the cleanliness of the room ten minutes ago and now," the planning intelligence analysis determines that its core operation is "comparison," and the comparison is with "the state ten minutes ago," which clearly relies on historical data. Therefore, it is determined that historical data needs to be acquired. For another subtask whose execution logic is "identifying the tool currently held by the user," the analysis points out that the condition of "currently held" must be confirmed through real-time visual perception. Therefore, it is determined that sensory data needs to be collected.

[0110] Based on the above judgment results, the task planning agent will generate input resource labels. If it is determined that historical data needs to be acquired, a historical data label will be generated and attached to the subtask; if it is determined that sensory data needs to be collected, a sensory data label will be generated.

[0111] Of course, these two labels are not mutually exclusive; a subtask may be labeled as requiring both. For example, a subtask with the execution logic of "according to the previous operation steps, now install part A onto the base being assembled" may require both historical data to confirm that the "previous operation steps" have been completed and real-time sensing data to locate the precise position of the "base being assembled".

[0112] If, after evaluation, it is determined that the execution of a subtask is completely independent of historical or real-time data from the external environment, then the planning agent does not need to generate any input resource labels for it. For example, a subtask that is purely data computation, acquiring data that is the output of an upstream subtask, does not require input resource labels.

[0113] Through the judgment and labeling process, each subtask carries a clear "list" of data requirements, i.e., input resource labels, before entering the execution queue.

[0114] Figure 2 This is an architecture diagram of multi-agent interaction provided in the embodiments of this specification. As can be seen, the environment memory agent converts video data into environment memory units for storage. The scheduling agent, as the core agent, calls other agents. The intent recognition agent provides the classification of user intent to determine whether to call the task planning agent (which performs complex task decomposition) or utilize the dialogue routing agent to determine the subdivided dialogue agents to execute the task. Simultaneously, when the task planning agent or various dialogue agents require historical data, the scheduling agent retrieves the data from the memory retrieval agent; or when sensing data is needed, it is obtained from the wearable device. Finally, the text dialogue agent determines the response information and returns it to the wearable device.

[0115] In the embodiments of this specification, both dialogue agents and task planning agents need to support reasoning and analysis of continuous events to make accurate decisions. However, traditional model training processes are insufficient in training models or agents to understand continuous events and perform temporal reasoning. Training methods based on static data and label mapping make it difficult for models to learn the crucial event evolution reasoning ability during task execution, i.e., understanding how the current state is caused by a series of historical actions and environmental changes, and planning the next step accordingly. This lack of ability directly leads to agents making one-sided and unreliable decisions when faced with complex tasks that require multi-step collaboration or rely on historical context.

[0116] Therefore, this specification also provides a training method for planning and generating agents that perform sequential subtasks. This training process combines a two-stage training strategy of supervised imitation and reinforcement learning optimization.

[0117] Specifically, first, the server acquires a large amount of pre-recorded guided task video data, such as cooking tutorials or furniture assembly tutorials.

[0118] Each of these videos contains visual scene changes that unfold in chronological order, accompanied by synchronized guided action instructions that guide the user through the steps.

[0119] The server can invoke a pre-defined, powerful teacher model to analyze video data. This teacher model's task is to automatically identify keyframes in the video that correspond to critical operations or state transitions, and to annotate each keyframe with a textual description of its corresponding current environmental state, as well as the guiding action instruction that should be executed at that moment.

[0120] For example, in a video of assembling a chair, the teacher model might identify "all screws are in place" as a keyframe state, with the corresponding guiding action instruction being "please pick up the screwdriver".

[0121] Subsequently, based on the structured annotations provided by the teacher big data model, the server constructs a reasoning chain for each keyframe as a supervision signal for that keyframe. For each selected keyframe, the teacher big data model not only identifies the current environmental state in the scene (e.g., "the pan is hot, the oil has been poured in"), but more importantly, it needs to review and reason about the key historical event chain that led to this state, and encapsulate this reasoning in a structured... <memory>Within the tags. At the same time, the teacher's big model provides guidance instructions for the action to be performed at this time based on the current state and history (e.g., "Add the chopped green onions").

[0122] In the example provided in this manual, the format of this thought chain is as follows: keyframe + <memory>Historical Event Reasoning <memory>→Current environment status analysis→Next guided action instruction. For example, <memory> The user has turned on the stove, heated the pot for about one minute, and then poured in cooking oil.< / memory> →Current status: The pan is heated, and the oil has been poured in and spread to cover the bottom of the pan. →Next step: Add the chopped green onions.

[0123] After obtaining a large amount of training data featuring keyframe-reasoning thought chains, the training transitions to supervised training. The server uses keyframes, or short video clips centered around them, as input to the agent, and the corresponding complete reasoning thought chains as training targets, to fine-tune the agent. The goal of this stage is to enable the agent to initially learn to imitate the reasoning patterns of the teacher's large model and master the ability to generate this structured thinking process that includes historical review.

[0124] Subsequently, after the agent learns to output thought chains containing historical information, reinforcement learning can be performed. The server, based on complete guided task video data, allows the supervised-trained agent to learn through trial and error in a simulation environment closer to real-world applications. The agent inputs a video stream and needs to output predicted action commands at different times. At this point, the reinforcement learning algorithm begins: its reward function is based on the difference (or matching degree) between the agent's predicted action command and the correctly identified guided action command labeled by the teacher's large model at that moment. By maximizing the cumulative reward, the agent is driven to adjust its internal parameters so that its generated predicted action commands not only mimic the form in the supervised phase but also pursue substantial accuracy and applicability within the complete task context during the reinforcement learning phase.

[0125] Through two phases of training, the agent will acquire a deep understanding of event evolution and robust sequential task planning capabilities.

[0126] Furthermore, to improve the effectiveness of reinforcement learning and address the problem of reward sparsity, deterministic event understanding quality rewards can be implemented. In complex sequence tasks, an agent may accidentally output a correct action instruction, but its reasoning process leading to this conclusion may have serious flaws, such as incorrectly attributing to historical events or establishing illogical connections between the current situation and the action. If only the correctness of the final action is used as the training signal, such erroneous reasoning patterns cannot be corrected, resulting in low robustness of the decision model learned by the agent.

[0127] Therefore, a reward function is provided that can evaluate and reward the quality of the agent's thinking process, enabling the agent to learn to generate reasoning chains that are both consistent with real history and have internal logical consistency.

[0128] Specifically, at each step of reinforcement learning training, after the agent outputs a complete reasoning for a keyframe that includes a review of historical events, analysis of the current environmental state, and prediction of action instructions, the reasoning process and the related training data are fed into the reward model for evaluation.

[0129] First, the matching degree of the historical event descriptions identified during the agent's reasoning process is evaluated. The reward model compares the historical events recalled by the agent with the actual sequential events (recorded by other keyframes) annotated by the teacher's large model. It determines whether the agent accurately identifies and summarizes the core preceding event chain that led to the current state. For example, if the actual history is "the user turned on the stove and waited 30 seconds," while the agent recalls "the user just washed the pots and pans," the matching degree is low.

[0130] Secondly, the reward model evaluates the logical coherence between the current environmental state determined by the agent and the predicted action instruction. It analyzes whether there is a reasonable causal relationship between the current state described by the agent and its suggested next action. For example, if the agent analyzes the current state as "the pot is cold and empty," but its predicted action is "to scoop the food out and plate it," then this state-action pairing is considered to lack logical coherence.

[0131] Based on a comprehensive analysis of the two dimensions above, the reward model calculates a specific reward value for the quality of event understanding. Specifically, the higher the degree of matching between the historical events recalled by the agent and the actual event chain, the greater the reward contribution; and the stronger the logical coherence between the inferred state and the suggested action, the greater the reward contribution.

[0132] Subsequently, this reward value, as a parallel item to the final action correctness reward, is incorporated into the overall reinforcement learning reward function to guide the agent's parameter updates. Through the continued operation of this mechanism, the agent is explicitly guided to optimize its internal reasoning process, learning to generate decision sequences that are not only correct in outcome but also rational and reliable in process.

[0133] Figure 3 This is a schematic diagram of the training process provided in the embodiments of this specification. It involves adding corresponding historical keyframes to video data to construct a novel thought chain for supervised fine-tuning (SFT). Then, in the reinforcement learning phase, by introducing rewards for the correctness of event reasoning, the model is guided to optimize both decision-making and its internal event evolution reasoning capabilities.

[0134] The above describes an interaction method for a wearable device provided in the embodiments of this specification. Based on the same idea, this specification also provides corresponding devices, storage media, and electronic devices.

[0135] Figure 4 This specification provides a schematic diagram of an interactive device for a wearable device, comprising: The environment memory module 401 is used to analyze and process the video data continuously acquired by the image sensor according to a preset frequency to obtain a number of environment memory units divided according to time sequence. The task planning module 402 is used to respond to user interaction requests, plan and generate serialized subtasks to be executed, and input resource tags required for the execution of the subtasks; The demand determination module 403 is used to schedule the input resource tag data required for the execution of each subtask according to the execution sequence of each subtask. The scheduled data includes historical data retrieved from the environment memory unit and / or sensing data collected from the sensors of the wearable device. Task execution module 404 is used to execute the subtask based on the data scheduled for the subtask; The interaction module 405 is used to generate and return response information in natural language description based on the execution results of each subtask and the interaction request when each subtask is completed.

[0136] Optionally, the environment memory module 401 is used to sample the video data continuously acquired by the image sensor at a preset frequency through a preset environment memory agent; analyze and process the acquired video data segments to determine the scene semantic information and scene summary of each video data segment; encapsulate the timestamp of the video data segment, the scene semantic information and the scene summary to determine the environment memory unit; and store the determined environment memory unit according to the timestamp.

[0137] Optionally, the planning task module 402 is used to send the interaction request to a preset intent recognition agent through a preset scheduling agent; the intent recognition agent performs semantic recognition on the interaction request to determine the user intent and intent category, the intent category including at least execution intent and dialogue intent; the scheduling agent calls the corresponding type of agent based on the intent category returned by the intent recognition agent, and plans and generates a serialized subtask to be executed and the input resource tags required for the execution of the subtask based on the user intent.

[0138] Optionally, the task planning module 402 is configured to, when the intent category is an execution intent, call a preset task planning agent to send the user intent to the task planning agent; through the task planning agent, determine the task to be executed corresponding to the user intent, and split the task to be executed into a plan to obtain a serialized subtask to be executed; for each subtask, generate structured execution logic and label the input resource tags required for the execution of the subtask.

[0139] Optionally, the planning task module 402 is configured to, when the intent category is an execution intent, call a preset dialogue agent to send the user intent to the dialogue agent; through the dialogue agent, generate a sub-task to be executed according to the user intent, the sub-task having structured execution logic and labels indicating the input resources required for the execution of the sub-task.

[0140] Optionally, the planning task module 402 is used for the scheduling agent to call a preset dialogue routing agent to send the user intent to the dialogue routing agent; through the dialogue routing agent, to determine the dialogue channel corresponding to the user intent; and to determine an agent that matches the dialogue channel from video dialogue agents, image-text dialogue agents, and text dialogue agents as the target agent; and the scheduling agent sends the user intent to the target agent.

[0141] Optionally, the requirement determination module 403 is used to determine, according to the execution logic, whether it is necessary to obtain historical data related to the subtask from the environment memory unit, and whether it is necessary to call the sensors of the wearable device to collect perception data; if it is determined that historical data needs to be obtained, a historical data tag is generated; if it is determined that perception data needs to be obtained, a perception data tag is generated; otherwise, it is determined that no input resource tag needs to be generated.

[0142] Optionally, the demand determination module 403 is configured to, when the input resource label of the subtask is a historical data label, send the user intent to a preset memory retrieval agent; and through the memory retrieval agent, retrieve the environmental memory unit related to the subtask from the stored environmental memory units as the historical data to be scheduled.

[0143] Optionally, the requirement determination module 403 is configured to, when the input resource label of the subtask is a perception data label, determine the perception data type based on the perception data label; determine the sensor corresponding to the perception data type from the sensors of the wearable device, and obtain the perception data collected by the sensor.

[0144] Optionally, the task execution module 404 is used to send the subtask scheduling data obtained by the scheduling agent to the corresponding type of agent, so that the corresponding type of agent can execute the subtask.

[0145] Optionally, the interaction module 405 is used for the scheduling agent to send the execution results of each sub-task and the user intent to the text dialogue agent; and through the text dialogue agent, to generate response information described in natural language and return the response to the user.

[0146] Optionally, the device further includes: a training module 406; The training module 406 is used to acquire pre-recorded video data of guided tasks, wherein the video data contains at least one guided action instruction, which is used to guide the user to perform an operation; using a preset teacher big model, it identifies keyframes in the video data, the current environmental state of the keyframes, and the correspondence between the keyframes and the guided action instructions; for each keyframe, it constructs an inference thought chain based on the keyframe, other keyframes preceding the keyframe in time sequence, the guided action instruction corresponding to the keyframe, and the current environmental state of the keyframe, as the annotation corresponding to the keyframe; using a small portion of the video data determined by the keyframes as input, and using the annotations corresponding to the keyframes, it performs supervised training on the agent; based on the video data, it uses reinforcement learning to further train the agent after supervised training, wherein the function value of the reward function of the reinforcement learning is determined based on the difference between the predicted action instruction output by the agent and the guided action instruction in the annotation.

[0147] Optionally, the training module 406 is configured to, for each keyframe, identify, through a preset reward model, the degree of matching between the historical events determined by the agent during the reasoning process of that keyframe and other keyframes preceding that keyframe in terms of time sequence; identify the logical coherence between the current environmental state of the keyframe determined by the agent and the predicted action instruction; and determine an event understanding quality reward as a term in the reward function based on the degree of matching and the coherence; wherein the degree of matching is positively correlated with the event understanding quality reward, and the coherence is positively correlated with the event understanding quality reward.

[0148] This specification also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can be used to perform the interaction method of the wearable device described above.

[0149] based on Figure 1 The interaction method of the wearable device shown in this specification is further provided in the embodiments. Figure 5 The diagram shows the structure of the electronic device. Figure 5 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the interaction method of the wearable device described above.

[0150] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.< / memory> < / memory> < / memory>

Claims

1. An interaction method for a wearable device, wherein the sensors of the wearable device include at least an image sensor and an audio sensor, the method comprising: The video data continuously acquired by the image sensor is analyzed and processed according to a preset frequency to obtain several environmental memory units divided according to time sequence. In response to user interaction requests, a serialized subtask to be executed is planned and generated, along with the input resource tags required for the execution of the subtask; According to the execution sequence of each subtask, for each subtask, the required input resource tag scheduling data is based on the subtask execution, and the scheduling data includes historical data retrieved from the environmental memory unit and / or sensing data collected from the sensors of the wearable device; Execute the subtask based on the data scheduled for it. When each subtask is completed, a response message in natural language is generated and returned based on the execution results of each subtask and the interaction request.

2. The method as described in claim 1, wherein the video data continuously acquired by the image sensor is analyzed and processed according to a preset frequency to obtain a plurality of environmental memory units divided according to time sequence, specifically including: The image sensor continuously collects video data through a preset environmental memory agent at a preset frequency. The acquired video data segments are analyzed and processed to determine the scene semantic information and scene summary of each video data segment; The timestamp of the video data segment, the scene semantic information, and the scene summary are encapsulated to determine the environment memory unit; The environmental memory unit is stored and determined according to the timestamp.

3. The method as described in claim 1, which involves planning and generating serialized subtasks to be executed, and the input resource tags required for the execution of the subtasks, specifically includes: The interaction request is sent to the preset intent recognition agent through a preset scheduling agent; The intent recognition agent performs semantic recognition on the interaction request to determine the user's intent and intent category, wherein the intent category includes at least execution intent and dialogue intent; The scheduling agent, based on the intent category returned by the intent recognition agent, calls the corresponding type of agent, and plans and generates a serialized subtask to be executed, as well as the input resource tags required for the execution of the subtask, based on the user intent.

4. The method as described in claim 3, wherein the scheduling agent, based on the intent category returned by the intent recognition agent, invokes an agent of the corresponding type, and based on the user intent, plans and generates a sequenced subtask to be executed, and input resource labels required for the execution of the subtask, specifically including: When the intent category is an execution intent, the scheduling agent invokes a preset task planning agent to send the user intent to the task planning agent; The task planning agent determines the task to be executed corresponding to the user's intent, and breaks down the task to be executed into a series of sub-tasks to be executed. For each subtask, generate structured execution logic and label the input resources required for the execution of that subtask.

5. The method as described in claim 3, wherein the scheduling agent, based on the intent category returned by the intent recognition agent, invokes an agent of the corresponding type, and based on the user intent, plans and generates a sequenced subtask to be executed, and input resource labels required for the execution of the subtask, specifically including: When the intent category is an execution intent, the scheduling agent invokes a preset dialogue agent to send the user intent to the dialogue agent; The dialogue agent generates subtasks to be executed based on the user's intent. The subtasks contain structured execution logic and labels indicating the input resources required for their execution.

6. The method as described in claim 5, wherein the scheduling agent invokes a preset dialogue agent to send the user intent to the dialogue agent, specifically including: The scheduling agent invokes a preset dialogue routing agent to send the user's intent to the dialogue routing agent; The dialogue routing agent determines the dialogue channel corresponding to the user's intent. And from video dialogue agents, image-text dialogue agents, and text dialogue agents, determine the agent that matches the dialogue channel as the target agent; The scheduling agent sends the user's intent to the target agent.

7. The method as described in claim 4 or 5, specifically including labeling the input resource tags required for the execution of the subtask, including: Based on the execution logic, it is determined whether it is necessary to obtain historical data related to the subtask from the environmental memory unit, and whether it is necessary to call the sensors of the wearable device to collect sensing data. If it is determined that historical data needs to be obtained, then a historical data label is generated; if it is determined that perceptual data needs to be obtained, then a perceptual data label is generated; otherwise, it is determined that no input resource label needs to be generated.

8. The method as described in claim 7, wherein the input resource tag scheduling data required for the execution of the subtask specifically includes: When the input resource label for the subtask is a historical data label, the scheduling agent sends the user intent to the preset memory retrieval agent; The memory retrieval agent retrieves the environmental memory units related to the subtask from the stored environmental memory units, as historical data that needs to be scheduled.

9. The method as described in claim 7, wherein the input resource tag scheduling data required for the execution of the subtask specifically includes: When the input resource label for the subtask is a perception data label, the scheduling agent determines the type of perception data based on the perception data label. From the sensors of the wearable device, determine the sensor corresponding to the type of sensing data, and obtain the sensing data collected by the sensor.

10. The method of claim 3, wherein the subtask is executed based on the data scheduled for the subtask, specifically comprising: The scheduling agent will send the acquired data of the subtask scheduling to the corresponding type of agent, which will then execute the subtask.

11. The method as described in claim 6, wherein, based on the execution results of each subtask and the interaction request, a response message describing natural language is generated and returned, specifically including: The scheduling agent sends the execution results of each sub-task and the user intent to the text dialogue agent; The text-based dialogue agent generates a response message in natural language and returns the response to the user.

12. The method of claim 3, wherein the agent for planning and generating sequential subtasks to be executed is trained using the following method, wherein: Acquire pre-recorded video data of guided tasks, wherein the video data contains at least one guided action instruction, which is used to guide the user to perform an operation; Using a pre-defined teacher model, keyframes in the video data, the current environmental state of the keyframes, and the correspondence between the keyframes and the guiding action instructions are identified. For each keyframe, a reasoning chain is constructed based on the keyframe, other keyframes preceding it in time sequence, the guiding action instructions corresponding to the keyframe, and the current environmental state of the keyframe, which serves as the annotation for the keyframe. Using the keyframes to determine a small portion of video data as input, and utilizing the annotations corresponding to the keyframes, the agent is subjected to supervised training. Based on the video data, reinforcement learning is used to further train the agent that has undergone supervised training. The reward function value of the reinforcement learning is determined according to the difference between the predicted action instructions output by the agent and the guided action instructions in the annotation.

13. The method of claim 12, further comprising: For each keyframe, a preset reward model is used to identify the historical events determined by the agent during the reasoning process of that keyframe, and the degree of matching between them and other keyframes preceding that keyframe in terms of time sequence. Identify the logical coherence between the current environmental state of the keyframe determined by the intelligent agent and the predicted action instructions; Based on the degree of matching and the coherence, an event comprehension quality reward is determined as one of the items in the reward function; The degree of matching is positively correlated with the event understanding quality reward, and the coherence is positively correlated with the event understanding quality reward.

14. An interaction device for a wearable device, wherein the sensors of the wearable device include at least an image sensor and an audio sensor, comprising: The environment memory module is used to analyze and process the video data continuously acquired by the image sensor according to a preset frequency to obtain a number of environment memory units divided according to time sequence. The task planning module is used to respond to user interaction requests, plan and generate serialized subtasks to be executed, and the input resource tags required for the execution of the subtasks; The demand determination module is used to schedule the input resource tag data required for the execution of each subtask according to the execution sequence of each subtask. The scheduled data includes historical data retrieved from the environment memory unit and / or sensing data collected from the sensors of the wearable device. The task execution module is used to execute the subtask based on the data scheduled for it. The interaction module is used to generate and return response information in natural language based on the execution results of each subtask and the interaction request when each subtask is completed.

15. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any one of claims 1-13.

16. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method of any one of claims 1-13.

Images