A cross-device task migration method, system, device and storage medium

Abstract

The application discloses a cross-device task migration method, system, device and storage medium. The method comprises the following steps: starting a target interactive task through a first interactive device in a first interactive scene, and maintaining execution state data of the target interactive task locally on a master device; in response to a scene switching trigger event, if the target interactive task is in an unfinished state, determining a second interactive device in a second interactive scene, and performing a connection permission check on the second interactive device; after the connection permission check is passed, generating connection interactive data based on an unfinished part of the target interactive task in the execution state data, and sending the connection interactive data to the second interactive device, so as to continue the human-computer interaction process of the target interactive task through the second interactive device. The present scheme realizes safe and seamless connection in the cross-device task migration process, and improves the coherence and intelligent level of the operation experience of the user in the multi-device collaborative scene.

Description

Technical Field

[0001] This invention relates to the field of multi-device interaction technology, and in particular to a method, system, device, and storage medium for cross-device task migration. Background Technology

[0002] Currently, users often need to switch between different interactive devices to complete the same interactive task in their daily use. During the development of this invention, the inventors discovered that existing technologies suffer from at least the following problems: Existing cross-device task migration methods generally lack a cross-device task state sharing and collaboration mechanism. When a user moves from their current interactive device to another device, the progress of the task being executed on the original device, the input parameters, and intermediate results cannot automatically follow the user's migration. This forces the user to re-execute the task from the beginning on the new device, causing not only repetitive operations and wasted time but also loss of task context, severely disrupting the continuity of operations in multi-device collaborative scenarios. Furthermore, existing systems lack a mechanism to verify the legitimacy and permissions of the target device when facing cross-device transfers, posing a security risk that task data may be obtained by unauthorized devices or irrelevant users. Summary of the Invention

[0003] This invention provides a cross-device task migration method, system, device, and storage medium to solve the problems of poor operational continuity and data security risks in existing cross-device task migration methods.

[0004] According to one aspect of the present invention, a cross-device task migration method is provided, the method comprising: The target interaction task is initiated through the first interaction device in the first interaction scenario, and the execution status data of the target interaction task is maintained locally on the main control device. In response to a scene switching event, if the target interaction task is in an incomplete state, the second interaction device in the second interaction scene is determined, and the connection permission is verified for the second interaction device. After the permission verification is passed, the follow-up interaction data is generated based on the unfinished part of the target interaction task in the execution status data, and the follow-up interaction data is sent to the second interaction device to continue the human-computer interaction process of the target interaction task through the second interaction device.

[0005] According to another aspect of the present invention, a cross-device task migration system is provided, the system comprising: The task initiation and status maintenance module is used to initiate the target interaction task through the first interaction device in the first interaction scenario, and to maintain the execution status data of the target interaction task locally on the main control device. The migration triggering and permission verification module is used to respond to scene switching trigger events. If the target interaction task is in an incomplete state, it determines the second interaction device in the second interaction scenario and performs a succession permission verification on the second interaction device. The task continuation module is used to generate continuation interaction data based on the incomplete part of the target interactive task in the execution status data after the continuation permission verification is passed, and send the continuation interaction data to the second interactive device so as to continue the human-computer interaction process of the target interactive task through the second interactive device.

[0006] According to another aspect of the present invention, a master control device is provided, the master control device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the cross-device task migration method according to any embodiment of the present invention.

[0007] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the cross-device task migration method according to any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the cross-device task migration method described in any embodiment of the present invention.

[0009] The technical solution of this invention maintains task execution status data locally on the main control device, ensuring that the core progress and parameters of the task do not depend on the local storage of any interactive device. This achieves centralized and persistent management of the task status on the main control device. When a scene switching event occurs, the main control device automatically determines that the task is incomplete and intelligently selects a suitable second interactive device. After permission verification, it accurately generates follow-up interactive data based on the locally stored incomplete portion and sends it to the second device. This eliminates the need for users to re-initiate or repeat operations when switching from the first interactive device to the second interactive device. The task context and execution progress are completely preserved and automatically continued, significantly improving the continuity and automation of task migration in multi-device scenarios. Simultaneously, mandatory verification of follow-up permissions before migration effectively prevents task data from being continued by unauthorized devices or irrelevant users, ensuring data security during the migration process. Furthermore, since core decisions and state storage are all concentrated on the main control device, the interactive devices only perform basic input / output functions, lowering the hardware threshold for lightweight terminals to participate in collaborative interaction and achieving uninterrupted, non-repetitive, secure, and controllable automatic migration of cross-device task execution.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart of a cross-device task migration method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the task initiation and status maintenance process provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the migration triggering and permission verification process provided in an embodiment of the present invention; Figure 4 A schematic diagram of the task continuation process provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a cross-device task migration system provided in an embodiment of the present invention; Figure 6 A timing diagram of a cross-device task migration method provided in an embodiment of the present invention; Figure 7This is a schematic diagram of the structure of a multi-terminal interactive system based on a multi-agent collaborative architecture, provided in an embodiment of the present invention. Figure 8 A schematic diagram of the structure of the master control device for implementing the cross-device task migration method of this embodiment of the invention. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0014] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0015] Figure 1 This is a flowchart illustrating a cross-device task migration method provided in an embodiment of the present invention. This embodiment is applicable to situations where, under the control of a main control device, an unfinished interactive task executed by a user through a first interactive device is migrated to a second interactive device to continue the human-computer interaction process, thereby achieving seamless continuation of cross-device tasks. This method can be executed by a cross-device task migration system, which can be implemented in hardware and / or software and can be configured in the main control device. Figure 1 As shown in the figure, this embodiment provides a cross-device task migration method, applied to the main control device, which specifically includes the following steps: S110. Start the target interaction task through the first interaction device in the first interaction scenario, and maintain the execution status data of the target interaction task locally on the main control device.

[0016] The main control device can refer to the terminal entity that serves as the sole decision-making center, task management, semantic parsing, and core data storage function throughout the entire task migration process. For example, the main control device can be a smartphone, tablet, or other device that the user carries with them and has strong computing power and a secure environment.

[0017] The first interactive scenario can refer to the interactive scenario in which the user is currently located before the task migration occurs, and which is used to initiate the target interactive task. It assumes the role of the scenario source for the migration. This scenario has the physical or functional conditions to deploy the corresponding interactive device. For example, the first interactive scenario can be an area such as a family living room, bedroom, vehicle cabin, or outdoor walkway. It should be understood that the specific physical location of this scenario is not limited, as long as the corresponding first interactive device is configured for the user to initiate the interaction. The first interactive device can refer to the terminal device used to initiate the target interactive task in the user's current interactive scenario before the task migration occurs. It assumes the function of the human-computer interaction entry point between the user and the main control device before the migration. This device has input acquisition (such as microphone, camera) and output feedback (such as display screen, speaker / bone conduction) capabilities. For example, the first interactive device can be a physical display screen attached to a wall, augmented reality (AR) glasses worn by the user, a smart speaker, or a vehicle central control screen. It is important to understand that this first interactive device does not perform core agent processing logic such as semantic understanding, logical decision-making, task state maintenance, or migration determination. It does not operate as an independent agent but only serves as an input / output peripheral for the main control device. It should also be clarified that this device plays the role of the migration source in this migration process.

[0018] A target interaction task can refer to a digital transaction unit triggered by a user through any interaction method, possessing a clear business intent and a complete execution process. For example, a target interaction task could be a restaurant query task, a schedule creation task, a weather query task, a smart home control task, or a file transfer task. The triggering methods for a target interaction task can include, but are not limited to, voice commands, touch operations, swipe operations, gesture recognition, and eye tracking. Execution status data can refer to a structured data set used to completely record the running status of the target interaction task, serving as the core benchmark for task progress tracking, breakpoint recovery, and cross-device continuity. For example, execution status data can include, but is not limited to, task identifier, task execution progress, task progress snapshot, task parameters, and task context.

[0019] In this embodiment of the invention, a user in a first interactive scenario can initiate an interaction request to a first interactive device. After receiving the request, the first interactive device will preprocess the original interactive signal. For example, it can perform sound source localization, echo cancellation, environmental noise reduction, and voice activity detection operations on the audio signal collected by the microphone array, coordinate smoothing filtering, accidental touch rejection, and trajectory compression operations on the touch signal collected by the touch screen, and pupil localization, gaze point stabilization, and noise filtering operations on the eye movement signal collected by the eye tracking module, so as to extract effective user intent expression from the original interactive signal and filter out environmental interference and invalid noise. Then, the preprocessed interactive request data is reported to the main control device.

[0020] Next, the main control device can parse key information such as user identity information, device identifier, interaction timestamp, and preprocessed valid signal stream from the interaction request data, and perform semantic parsing and intent recognition. For example, for the valid signal stream of the voice modality, the main control device can call the automatic speech recognition module to convert it into standardized text, and then use the natural language understanding module to classify intent and fill slots. For the valid signal stream of the touch or swipe modality, the main control device can use the touch intent mapping module to map features such as touch trajectory coordinates, swipe direction, and operation duration into corresponding operation instructions or intent labels, thereby identifying the user's interaction intent and starting the corresponding target interaction task. After the task is started, the main control device will generate a globally unique task identifier for the target interaction task locally, and create structured execution status data containing key information such as task identifier, task execution progress, task progress snapshot, task parameters, and task context. During the subsequent execution of the target interactive task, the master control device can respond to subsequent interactive commands input by the user through the first interactive device in any interactive manner, and continuously update the corresponding fields in the execution status data, so as to maintain the latest running status of the target interactive task locally on the master control device in real time, and provide an accurate data benchmark for possible subsequent cross-device task migration.

[0021] S120. In response to the scene switching trigger event, if the target interaction task is in an incomplete state, determine the second interaction device in the second interaction scene and perform a succession permission verification on the second interaction device.

[0022] Scene switching trigger events can refer to various trigger signals that can characterize changes in the user interaction scene and meet preset migration start conditions. These may include, but are not limited to, user departure events detected by the interactive device, user-initiated task migration commands, device location status change signals, and preset scene timed switching signals.

[0023] The second interaction scenario refers to another interaction scenario that the user intends to switch to during task migration, used to continue the target interaction task. It assumes the role of the migration target scenario. This scenario also possesses the physical or functional conditions for deploying corresponding interaction devices. For example, the second interaction scenario can be an outdoor street, a vehicle cabin, an office, or a family living room. It should be understood that this scenario is not fundamentally limited in terms of physical space type to the first interaction scenario; the two can be interchanged (e.g., the first scenario is indoors and the second scenario is outdoors, or vice versa). The distinction is based solely on the user's spatiotemporal location during the migration process, and both scenarios must be configured with interaction devices controlled by the main control device.

[0024] The second interaction device can refer to the terminal device that the user intends to switch to in another interaction scenario during task migration, used to continue the target interaction task. It serves as the human-computer interaction entry point between the user and the main control device after the migration. This device possesses the same type of input acquisition and output feedback capabilities as the first interaction device. Examples include AR glasses worn by the user, a physical display screen attached to a wall, a portable projector, and a vehicle central control screen. It is important to understand that this second interaction device does not perform core agent processing logic such as semantic understanding, logical decision-making, task state maintenance, or migration determination. It merely acts as an input / output peripheral for the main control device, replacing the first interaction device as the human-computer interaction entry point between the user and the main control device after the migration is complete; that is, it plays the role of the migration target.

[0025] Succession permission verification can refer to the multi-dimensional legality and compliance verification process performed by the master control device on the selected second interactive device. It is used to confirm whether the interactive device is qualified to undertake the human-computer interaction process of the target interactive task. It is a core link to ensure the security of task flow. For example, succession permission verification may include, but is not limited to, device binding relationship verification, task ownership verification, device interaction permission verification, migration compliance verification, etc.

[0026] In this embodiment of the invention, during the execution of the target interactive task, the master control device can continuously monitor various scene switching trigger signals. When a valid scene switching trigger event conforming to preset rules is captured, the current execution status of the target interactive task can be queried from the execution status data maintained in this update based on the task identifier carried by the event. If the task has reached the end node, the migration process is terminated, and only the event log is retained. If the task is still in an incomplete state, the task continuation process is executed. The master control device can, based on preset scene and device mapping rules, the running status information of currently online interactive devices, and combined with scene attribute matching logic, select devices belonging to the second interactive scene and possessing basic human-computer interaction capabilities within the scope of all interactive devices it can control, and determine them as the continuation second interactive device. After determining the second interactive device, the master control device can perform continuation permission checks on it, including but not limited to device binding relationship verification, task ownership verification, device interaction permission verification, and migration compliance verification, to prevent unauthorized device access and ensure end-to-end security of cross-scene migration.

[0027] S130. After the succession permission verification is passed, succession interaction data is generated based on the unfinished part of the target interaction task in the execution status data, and the succession interaction data is sent to the second interaction device to continue the human-computer interaction process of the target interaction task through the second interaction device.

[0028] The "incomplete portion" refers to the set of all logical nodes and data states within the complete task flow defined by the execution status data of the target interaction task that have not yet met the completion conditions or reached the final state. It is used to determine the precise starting anchor point for task resumption after cross-device migration, enabling the second interaction device to continue subsequent human-computer interaction from this anchor point, rather than starting the entire task from scratch. For example, the incomplete portion may include parameter slots to be filled, sub-task nodes to be executed, asynchronous operation states that have been initiated but not yet returned, and user confirmation steps to be performed.

[0029] Continuing interaction data can refer to the data set generated by the main control device based on the incomplete part of the target interaction task in the execution status data, which is used to drive the second interaction device to continue the human-computer interaction process. For example, it may include task identifiers, device handover certificates, continuation breakpoint datasets (such as progress nodes, filled parameters, dialogue history), interaction prompt information (such as display text, audio parameters), etc.

[0030] The human-computer interaction process that continues the target interaction task can refer to the process where a second interaction device replaces the first interaction device and becomes the new user interaction entry point for the target interaction task, undertaking the functions of presenting interactive content and collecting user input, so that the user can continue to complete the task interaction from the interruption point.

[0031] In this embodiment of the invention, after confirming that the second interactive device has passed the connection permission verification, the main control device can extract breakpoint data corresponding to the incomplete part of the target interactive task from the locally maintained execution status data. For example, it can extract the globally unique session ID from the task identifier; extract the identifier of the currently incomplete sub-task node and the completion mark of the completed sub-task from the task execution progress; extract the slot data filled at the time of interruption, the executed tool call status, and the generated intermediate results from the task progress snapshot; extract the complete intent category and the filled key-value pairs of key parameters from the task parameters; and extract the dialogue history of the most recent N rounds and the corresponding semantic vectors from the task context. Then, based on the extracted breakpoint data, the corresponding connection interaction data is generated and sent to the second interactive device so that the device can continue the human-computer interaction process of the target interactive task. Subsequently, the user's interactive input such as voice reply or gesture operation on the second interactive device is collected and preprocessed by the second interactive device and forwarded to the main control device for semantic parsing and task scheduling. The main control device continues to update the execution status data and sends the processing results back to the second interactive device for presentation, forming a complete interactive closed loop.

[0032] It is important to understand that in the entire task migration process of this embodiment of the invention, all core logic, including the initiation of the target interaction task, the maintenance of execution status data, the response to scene switching trigger events, the verification of succession permissions, the generation of succession interaction data, and the subsequent task scheduling and execution, always runs on the main control device. The main control device is the sole holder of the task status and the sole executor of migration decisions. The first and second interaction devices in the system architecture only serve as human-computer interaction entry points for users. Each is only responsible for the acquisition of multimodal input signals (such as audio, image, and posture sensing data) and the presentation of output signals (such as display rendering and audio playback). They do not participate in the maintenance and updating of the task execution status, nor do they perform any judgments or decisions regarding migration logic. During the migration process, the succession interaction data received by the second interaction device is only temporarily cached in its running memory for immediate presentation and does not change the architectural essence of the main control device as the sole trusted data base and the sole logic processing center. In other words, the interactive task always runs inside the main control device, while the human-computer interaction interface only switches between the first and second interactive devices. The cross-device task migration perceived by the user is actually an orderly switching of the interaction entry point by the main control device, rather than a transfer of the task processing entity or task state.

[0033] The technical solution of this invention maintains the execution status data of the target interactive task locally on the main control device, so that the task progress and context do not depend on any single interactive device. When a scene switching trigger event occurs, the main control device automatically determines that the task is incomplete and only sends the follow-up interactive data to the second interactive device that has passed the permission verification. This not only achieves seamless continuation of cross-device tasks and avoids the tedious experience of users having to start from scratch due to device switching, but also effectively prevents task data from being obtained by unauthorized devices or irrelevant users through the permission verification mechanism before migration. In multi-device collaborative scenarios, it takes into account both interaction continuity and data security.

[0034] Furthermore, based on the above embodiments of the invention, such as Figure 2 As shown, the task initiation and status maintenance process specifically includes: S210: Receive a user interaction request sent by the first interactive device and verify the user interaction request.

[0035] Among them, a user interaction request can refer to a request instruction initiated by a user through a first interaction device to a main control device, which carries the user's interaction needs and input information, and is used to trigger the main control device to start the corresponding interaction task.

[0036] In this embodiment of the invention, the master control device can receive a user interaction request sent by the first interaction device in the first interaction scenario. The request data may include user-input interaction information (such as voice audio stream or text) and device identifier of the first interaction device. Then, the master control device can perform multi-dimensional verification operations on the request, such as verifying the integrity of the data packet and verifying whether the device identifier belongs to a bound trusted device. Only after the verification is passed will the subsequent processing flow be entered, thereby intercepting illegal, invalid or unauthorized interaction requests.

[0037] S220. After the user interaction request is verified, semantic parsing and intent recognition are performed on the user interaction request to initiate the corresponding target interaction task.

[0038] Semantic parsing and intent recognition refer to the process by which the main control device parses the natural language or interactive instructions in the user's interaction request, extracts the core semantic information, and identifies the user's true interactive intent. This process can transform the user's natural expression into a structured task description, which may include the identified action type (such as query, control, setting) and the related parameter objects (such as time, location, people), thereby providing standardized input for subsequent task matching and execution.

[0039] In this embodiment of the invention, after the main control device confirms that the user interaction request has been verified, it can further perform semantic parsing and intent recognition on the request to extract the user's core interaction needs and business intent, and then start the corresponding target interaction task locally and allocate corresponding system resources to it.

[0040] S230. Generate and continuously update the execution status data of the target interactive task locally on the main control device; the execution status data includes at least: task identifier, task execution progress, task progress snapshot, task parameters, and task context.

[0041] The task identifier refers to a unique identifier assigned to the target interactive task, used to distinguish different interactive tasks in the multi-tasking environment of the main control device, ensuring the accuracy of task matching and status retrieval. Task execution progress refers to status information characterizing the current completion level of the target interactive task, such as task node number, completion percentage, and current execution steps, and can be updated in real time based on task execution status. A task progress snapshot is a frozen record of the complete running state of the target interactive task at a specific moment, containing all key status information of the task at that moment, such as the output of completed sub-steps, intermediate calculation results, and system runtime environment parameters, so that execution can continue from that breakpoint during migration or recovery without starting from the beginning.

[0042] Task parameters refer to the core business parameters and configuration parameters that support the execution of the target interactive task. These may include, but are not limited to, filled slots (user-provided instruction information), unfilled slots (missing key task information identified based on intent), task priority, timeout, maximum retries, dependent task identifiers, and execution permission levels. A slot can be understood as a predefined "information empty space" used to hold the key information necessary to complete the target interactive task. Both filled and unfilled slots together define the current data completeness of the task and the direction of subsequent interaction. For example, taking a restaurant reservation task, the system predefines two mandatory slots: "Time" and "Number of People." When a user indicates "Reserving a restaurant on Saturday," the system fills the "Saturday" slot (filled), but the "Number of People" slot remains empty (unfilled). At this point, the system can ask "How many people?" to guide the completion of the information. Only when both slots are filled can the task enter the final query and execution phase. As can be seen, the filled slots provide the currently available query conditions, while the unfilled slots indicate the direction in which the interaction continues. Together, they control the entire process of the task from information collection to final execution.

[0043] Task context can refer to historical interaction information, business context data, and environmental context information accumulated during the execution of the target interaction task. It is used to ensure the continuity of interaction when connecting across devices, so that subsequent interactions can be based on the previous dialogue background.

[0044] In this embodiment of the invention, once the target interactive task is initiated, the main control device immediately generates independent execution status data for the task in its local storage unit, assigns a globally unique task identifier to the task, and initializes basic status fields such as task execution progress, task parameters, and task context. During subsequent task execution, the main control device can synchronize and update the execution status data in real time based on the progress of task nodes, updates to user interaction input, and processing results of business logic, ensuring that the status data is always consistent with the actual running state of the task. Simultaneously, the main control device generates task progress snapshots at key task nodes, fixed periods, or preset trigger conditions, storing the snapshot data in the execution status data to record the complete task state at a specific moment, providing a baseline for subsequent task breakpoint recovery and state backtracking.

[0045] This embodiment achieves complete decoupling of business processing and interaction entry by centralizing the core execution logic and status data of the target interaction task on the main control device. The first interaction device only undertakes input and output functions and does not need to have complex task processing capabilities, thus reducing the hardware and computing power requirements of the terminal device. At the same time, the centralized execution status data maintenance mechanism ensures the uniqueness and consistency of the task status, avoids data deviation and synchronization latency caused by multi-terminal synchronization, and provides an accurate and unified status data foundation for seamless task migration and breakpoint continuation in subsequent cross-device scenarios, greatly improving the reliability and interaction continuity of cross-device task flow.

[0046] Furthermore, based on the above embodiments of the invention, such as Figure 3 As shown, the migration triggering and permission verification process specifically includes: S310, Receive the departure event and task progress snapshot reported by the first interactive device, and use the departure event as a scene switching trigger event; wherein, the departure event is generated by the first interactive device when it detects that the user has left its interactive area.

[0047] Among them, the departure event can refer to the notification signal generated and reported to the main control device after the first interactive device detects that the user has left its interactive coverage area through its own perception capabilities, which is used to indicate that the user has left the current first interactive scenario.

[0048] In this embodiment of the invention, the master control device can continuously monitor the reported information from the first interactive device with which it is communicatively connected. When the user leaves the interactive area of ​​the first interactive device, the first interactive device can detect the user's presence status in real time through its sensors. When it is determined that the user has left the area, the first interactive device will generate a departure event and report it to the master control device. The master control device will then use the received departure event as the scene switching trigger event for this cross-device migration. In addition, when generating the departure event, the first interactive device will also save a snapshot of the task progress of the current target interactive task, which may include information such as completed slot filling data and dialogue round records. Then, it will report the task progress snapshot and the timestamp of the departure time to the master control device.

[0049] S320. Based on the task execution progress in the execution status data, determine whether the target interactive task is in an incomplete state.

[0050] In this embodiment of the invention, after receiving the departure event reported by the first interactive device, the master control device can extract the task execution progress field from the execution status data maintained locally to determine whether the target interactive task is currently in an incomplete state. If the task is determined to be in an incomplete state, the migration process is allowed to continue. If the task has been completed, the migration process is terminated and no further steps are required.

[0051] S330. If it is determined that the target interactive task is in an incomplete state, the interaction channel with the first interactive device is interrupted, and a task progress snapshot is written to the execution status data.

[0052] The interaction channel can refer to the logical communication link between the main control device and the interaction device, which is used to transmit human-computer interaction data and carry out task interaction processes.

[0053] In this embodiment of the invention, if the target interactive task is determined to be in an incomplete state, the main control device can interrupt its interaction channel with the first interactive device, terminating the task input acquisition and result output based on the first interactive device; simultaneously, it writes the task progress snapshot reported by the first interactive device into the local execution status data, locking the progress benchmark at the time of task interruption. In one embodiment, a distributed lock can also be applied to the execution status data to prevent concurrent modification by other devices during subsequent task migration, ensuring data consistency.

[0054] S340: Send device status query commands to each candidate interactive device that is connected to the main control device.

[0055] In this embodiment of the invention, the main control device can send a unified device status query command to each candidate interactive device that is connected to it, such as AR glasses, tablet computers, vehicle terminals, etc., to query the current operating status and task succession carrying capacity of each candidate interactive device.

[0056] S350: Receive the connection readiness status information sent by each candidate interactive device. The connection readiness status information is used to indicate whether each candidate interactive device has the conditions to continue the human-computer interaction process of the target interactive task.

[0057] Among them, the succession readiness status information can refer to the status feedback information fed back by the candidate interactive device to the main control device, which is used to characterize whether it currently has the human-computer interaction capability to undertake the target interactive task. For example, it can include the online status of the device, the wearing status, the hardware readiness status, etc.

[0058] In this embodiment of the invention, the main control device can receive connection readiness status information from each candidate interactive device to determine whether each device has the conditions to continue the human-computer interaction process of the target interactive task. In one embodiment, taking AR glasses as an example, the AR glasses can determine the wearing confidence level, whether the core interactive modules (such as microphone, speaker, and display screen) have completed hardware initialization, and whether the network latency is within the allowable range by fusing the configured infrared proximity sensor and gyroscope. Then, it encapsulates its self-test results into connection readiness status information and reports it to the main control device through an encrypted communication channel.

[0059] S360. Based on the connection readiness status information, determine one device from the candidate interaction devices that matches the second interaction scenario and meets the connection conditions as the second interaction device.

[0060] The succession conditions can refer to pre-configured judgment rules for measuring whether candidate interactive devices can effectively succeed the target interactive task. For example, the conditions may include, but are not limited to, the device must be online, have interactive hardware capabilities that meet the task requirements, and network latency must be within the allowable time limit.

[0061] In this embodiment of the invention, the main control device can integrate the connection readiness status information of each candidate device and select a device that matches the second interaction scenario and meets the connection conditions as the second interaction device for the connection target interaction task.

[0062] Furthermore, based on the above embodiments of the invention, S360 may include: Based on the connection readiness status information, candidate interaction devices that match the second interaction scenario and meet the connection conditions are identified as available devices. If there is only one available device, then the available device is designated as the second interactive device. If there are multiple available devices, one of them will be selected as the second interaction device according to a preset selection strategy.

[0063] The preset selection strategy can refer to a set of decision rules pre-configured within the main control device when there are multiple available devices, which are used to select a unique second interactive device. For example, the preset selection strategy may include a device priority strategy based on the user's historical usage preferences, a dynamic weighting strategy based on the real-time operating status parameters of each candidate device (such as signal strength and remaining power), and a spatial proximity strategy based on the current physical distance between the device and the user.

[0064] Specifically, the main control device can analyze the connection readiness status information reported by each candidate interactive device, extracting key information such as device type, hardware capabilities, current posture (e.g., whether it is being worn), network quality, and physical environment. Then, the main control device matches this information with preset second interactive scenario requirements. For example, if the second interactive scenario is "an outdoor mobile scenario where the user is about to go out," then fixed wall-mounted screens are excluded, and only portable wearable devices are retained. Simultaneously, the main control device checks whether each device meets preset connection conditions, such as, but not limited to, the device being online and fault-free, the core interactive module having been initialized, network latency below a preset threshold, and remaining battery power above a safety threshold. Only when a candidate interactive device meets both scenario matching and connection conditions are it marked as a usable device.

[0065] Next, the master control device can count the total number of currently marked as available devices. If this number equals 1, the master control device directly determines that available device as the final second interaction device, and then initiates a connection permission verification process for that device. If the number of available devices is greater than 1, meaning there are multiple candidate interaction devices that can adapt to the second interaction scenario and meet the connection conditions, the master control device will select one of them as the final second interaction device according to a preset selection strategy. For example, the master control device can calculate a comprehensive score for each available device based on its priority weight, signal strength, remaining battery power, processor load rate, and other information, and determine the device with the highest comprehensive score as the final second interaction device.

[0066] This embodiment of the solution introduces a dual admission mechanism of "scene matching" and "basic conditions," which effectively filters out invalid devices that are incompatible with the subsequent use environment or whose hardware status does not meet the standards, thus significantly improving the accuracy and success rate of migration target positioning.

[0067] S370. Perform a connection permission verification on the second interactive device. The connection permission verification includes at least one of the following: device binding relationship verification, task ownership verification, device interaction permission verification, and migration compliance verification.

[0068] Among these, device binding verification verifies whether the second interactive device has been legally bound to the current user beforehand, i.e., whether the device belongs to the user's authorized personal device ecosystem. Task ownership verification verifies whether the target interactive task to be migrated truly belongs to the user initiating the migration operation, preventing task interference or misoperation by multiple users in the same physical space. Device interaction permission verification verifies whether the second interactive device has the specific hardware or system permissions necessary to complete the subsequent steps of the task, such as microphone recording permissions and location access permissions. Migration compliance verification verifies whether the migration operation complies with preset data privacy and security policies, such as whether it complies with the system's preset data privacy protection policies, regional data localization requirements, or enterprise security compliance standards (e.g., ensuring that task data does not leave the user's local terminal during the migration process, or is not transferred across borders).

[0069] In this embodiment of the invention, after selecting the second interactive device, the master control device can perform a connection permission verification on it. The verification dimensions may include at least one of the following: device binding relationship verification, task ownership verification, device interaction permission verification, and migration compliance verification. Only if the above connection permission verification passes will the master control device continue to execute the subsequent task migration process; if the connection permission verification fails, the task migration process will be terminated, and the interaction channel will be switched back to the first interactive device.

[0070] This embodiment automatically triggers the migration process by receiving an exit event reported by the first interactive device. Based on the locally maintained task execution progress, it determines that the task is not completed, then interrupts the original interaction channel and saves a progress snapshot, ensuring the complete preservation of the task status. At the same time, by actively querying the connection readiness status of all candidate devices and matching and filtering them in combination with the target scenario, the most suitable second interactive device is accurately selected, avoiding invalid migration. Before the final handover, multi-level connection permission verification is performed on the selected device, thereby achieving seamless and reliable connection of cross-device tasks while ensuring the migration process is safe and controllable. This significantly improves the interactive experience and system robustness in multi-device collaborative scenarios.

[0071] Furthermore, based on the above embodiments of the invention, such as Figure 4 As shown, the task continuation process specifically includes: S410. After the connection permission verification is successful, a connection confirmation prompt instruction is sent to the second interactive device, and a user feedback instruction is received from the second interactive device.

[0072] The "continuation confirmation prompt" can be a control command sent from the main control device to the second interactive device, which triggers the device to display a confirmation prompt to the user to indicate their willingness to continue the task. The "user feedback" can be a command sent back from the second interactive device to the main control device after collecting user actions, which indicates the user's willingness to continue the task, including states such as confirming or rejecting the task.

[0073] In this embodiment of the invention, after confirming that the second interactive device has passed the connection permission verification, the main control device can send a connection confirmation prompt instruction to it to drive the second interactive device to display the task connection confirmation prompt to the user through its own interactive interface, and ask the user whether he / she agrees to migrate the human-computer interaction entry of the target interactive task to this device, while receiving the user feedback instruction returned by the second interactive device.

[0074] For example, taking AR glasses as an example, after receiving the connection confirmation prompt instruction from the main control device, the AR glasses can display a confirmation prompt bubble through the optical waveguide and simultaneously play a bone conduction prompt sound to ask the user whether they agree to the task transfer; then, the AR glasses open the gaze confirmation channel (such as eye tracking to detect the gaze confirmation area for more than 500 milliseconds and with a confidence level of ≥95%) and the voice confirmation channel (bone conduction microphone to collect and recognize affirmative keywords) in parallel. After either channel completes the confirmation first, the AR glasses immediately generate a "confirm connection" user feedback instruction and send it back to the main control device; if no confirmation is made within the time limit or a negative word is recognized, a "refuse connection" user feedback instruction is sent back to the main control device.

[0075] S420. If the user feedback command is to confirm the continuation, then generate task continuation breakpoint data based on the task identifier, task execution progress and task progress snapshot in the execution status data, and generate interactive prompt information based on the task parameters and task context in the execution status data.

[0076] Among them, task continuation breakpoint data can refer to the status data generated by the master control device based on the task identifier, task execution progress and task progress snapshot. It is used to enable the second interactive device to restore the technical execution state of the target interactive task (such as filled semantic slots, completed sub-step nodes and intermediate calculation results, etc.) from the breakpoint when continuating cross-device tasks, so as to ensure the accurate continuity of the task status after continuation.

[0077] Interactive prompts can refer to data generated by the main control device based on task parameters and task context, used to drive the second interactive device to present prompts to the user after connection. This data carries the contextual information required for the current task, supporting the second interactive device in inheriting the task interaction context, so that the user knows the next actionable interaction without actively asking, thereby ensuring the continuity of the interactive experience after cross-device connection. For example, the prompt content driven by the interactive prompts can be presented in various forms, including but not limited to voice broadcasting, text display, graphical interface, vibration feedback, holographic indication, and other interactive modalities.

[0078] In this embodiment of the invention, if the user feedback is determined to be a confirmation of continuation, the main control device will generate the following two types of key data or information based on the current execution status data: ① The main control device can locate the specific task based on the task identifier in the execution status data, and then extract the task execution progress and task progress snapshot (such as filled slot key-value pairs, dialogue round records, sub-step nodes at the time of interruption, etc.), integrate and encapsulate these structured status information to form the corresponding task continuation breakpoint data. The core function of this data is to enable the migrated task to be accurately restored to the technical execution scene before the interruption, without losing any completed operation results. ② Based on the task parameters (such as intent type, filled slots, etc.) and task context (such as containing complete task history and semantic vectors) in the execution status data, the main control device automatically infers the interaction steps that have not yet been completed when the current task is interrupted, and generates interactive prompt information to guide the user to continue to complete the steps.

[0079] For example, continuing with the restaurant reservation task, the main control device can extract the task parameters (intent: reservation, filled time and number of slots) and task context (the previous record where the system inquired about dietary restrictions) from the execution status data. After inference by the dialogue state tracking model, it accurately determines that the currently missing interaction step is that dietary restrictions or preferences have not yet been collected. Based on this, the main control device matches the query text "Do you have any dietary restrictions or preferred cuisines?" from the interaction template library, then calls the Text-to-Speech (TTS) system to convert the query text into an audio stream, and performs bone conduction EQ equalization on the audio to adapt to the output characteristics of the AR glasses. Finally, the adapted audio data is combined with the text to form a structured interactive prompt, which will then be sent to the AR glasses for display and voice playback, guiding the user to complete the missing interaction step.

[0080] S430. Generate continuity interaction data based on task continuity breakpoint data and interaction prompts.

[0081] In this embodiment of the invention, the main control device can fuse and encapsulate the aforementioned generated task continuation breakpoint data and interactive prompt information to form unified continuation interaction data. This data simultaneously includes the background state parameters required to restore the task execution context, as well as the foreground content displayed to the user, ensuring that the second interactive device, upon receiving it, can both reconstruct the task context and immediately present the next interactive content to the user.

[0082] S440. Switch the interaction channel of the target interaction task to the second interaction device, and send the follow-up interaction data to the second interaction device so that the second interaction device can continue the human-computer interaction process of the target interaction task.

[0083] In this embodiment of the invention, the master control device can formally switch the main interaction channel of the target interactive task from the first interactive device to the second interactive device, establishing the second interactive device as the sole human-computer interaction entry point for the current task. Simultaneously, through the established encrypted communication link, the aforementioned generated follow-up interaction data is sent to the second interactive device. Upon receiving the follow-up interaction data, the second interactive device aligns the task execution progress based on the task follow-up breakpoint data, renders and displays the task interaction interface based on the interaction prompts, and takes over the human-computer interaction flow of the target interactive task. Subsequent interaction data input by the user through the second interactive device will be transmitted back to the master control device, which will then continue executing the core business logic of the task and update the execution status data.

[0084] This embodiment effectively avoids the problem of accidental migration by employing a pre-confirmation user mechanism, ensuring users' control over task flow. Furthermore, the data to be sent is precisely broken down into "breakpoint data" for backend state recovery and "interactive prompts" for foreground interaction guidance. This ensures accurate reconstruction of the task execution context at the technical level and allows users to immediately receive natural next-step prompts after migration, achieving precise alignment of task progress and complete continuity of the interaction context. This provides users with a seamless and smooth cross-device transition experience. Simultaneously, the main control device maintains centralized control over the core task logic throughout the process, sending interaction layer data only to interactive devices. This ensures consistency of task status and business logic under multi-device switching and enhances data security and system stability in a multi-terminal collaborative architecture.

[0085] Furthermore, based on the above embodiments, the method further includes: If no user feedback instruction is received or the user feedback instruction is a refusal to continue, the current task migration process will be terminated. Send a device recovery command to the first interactive device. The device recovery command is used to instruct the first interactive device to restore the interactive state corresponding to the target interactive task, so as to switch the interactive channel of the target interactive task back to the first interactive device. Send a migration cancellation command to the second interactive device. The migration cancellation command is used to instruct the second interactive device to release the temporary interactive resources associated with this task migration.

[0086] The device recovery command can be a control command issued by the master control device to the first interactive device when it determines that the task migration has terminated. This command triggers the first interactive device to restore the interactive capabilities and access permissions corresponding to the target interactive task, thus resuming the interrupted task. The migration cancellation command can be a control command issued by the master control device to the second interactive device when it determines that the task migration has terminated. This command triggers the second interactive device to terminate the task migration preparation process and clean up temporary configurations and resources related to the task migration. Temporary interactive resources can refer to temporary hardware and software resources such as temporary storage space, interface rendering processes, interactive listening threads, and temporary configuration parameters pre-allocated by the second interactive device to handle this task migration. These resources serve only this migration process and are released after the migration terminates.

[0087] In this embodiment of the invention, the cross-device task migration method provided may further include the following fallback handling process: S1. After sending a connection confirmation prompt to the second interactive device, the main control device can simultaneously activate a timeout monitoring mechanism. If no user feedback instruction is received from the second interactive device within a preset time period, or if the received user feedback instruction indicates that the user has refused to continue the task, the migration is deemed to have failed to meet the execution conditions, and all subsequent processes of the migration are officially terminated, with no further migration operations such as connection data generation and channel switching performed. Furthermore, the main control device can update the record status of this migration operation to "cancelled" in its local execution status data, and record the cancellation reason (timeout or user refusal) and cancellation timestamp for subsequent auditing and problem tracing.

[0088] S2. The main control device can send a device recovery command to the first interactive device through an encrypted and trusted channel to instruct the first interactive device to restore the interactive output and input acquisition capabilities corresponding to the target interactive task, and switch the main interactive channel of the target interactive task back to the first interactive device, thereby restoring the normal human-computer interaction link in the first interactive scenario.

[0089] S3. The main control device can send a migration cancellation command to the second interactive device to instruct the second interactive device to remove the prompts related to task continuation, release the temporary interactive resources pre-allocated for this migration, end the migration preparation state, and thus return the device to its normal operating mode.

[0090] This embodiment, through the aforementioned fallback mechanism, endows the system with high fault tolerance and degradation capabilities in scenarios of user rejection or timeout failure to respond. Orderly transaction rollback ensures that the first interactive device can accurately restore its task state before the interruption, ensuring that the user's interactive experience on the original device is not negatively affected. Simultaneously, it forces the second interactive device to clear all temporarily cached task data and release hardware resources, eliminating the risk of residual user privacy on unauthorized devices from both physical and logical perspectives. This effectively protects data security and optimizes device energy consumption, significantly improving the engineering robustness and user trust of the multi-device collaborative system in real-world, complex usage environments.

[0091] Furthermore, based on the above embodiments of the invention, before initiating the target interactive task, the following is also included: Identify interactive devices within the preset communication range; Establish encrypted communication channels with each interactive device separately; Each interactive device is synchronized with its own clock.

[0092] The preset communication range can refer to the effective communication distance range that the main control device pre-sets based on the coverage capability of its own communication module and the spatial boundaries of the application scenario. Interactive devices within this range can establish a stable and reliable communication connection with the main control device.

[0093] In this embodiment of the invention, before the main control device starts the target interactive task, the following initialization process may also be included: S1. Before starting the target interaction task, the main control device can first start the device discovery process. Based on the wireless / wired communication protocols it supports, it scans all terminal nodes within the preset communication range, filters out interactive devices with human-computer interaction capabilities and support communication with the main control device, and forms a candidate interactive device set. This provides a basic device list for subsequent task interaction entry selection and cross-device migration target device screening.

[0094] For example, the master control device can send service discovery broadcasts to the currently connected local area network (LAN) via protocols such as Multicast Domain Name System (mDNS) and DNS-Based Service Discovery (DNS-SD). All terminal devices supporting these protocols and located on the same network segment (such as large-screen TVs, AR glasses, smart speakers, and in-vehicle terminals) must send a response with their device information to the master control device upon receiving the broadcast. This response information may include: the device's unique hardware identifier (such as a MAC address or device serial number), device type code, current firmware version number, and the set of interactive capabilities it supports (e.g., whether it has a microphone array, display screen, depth sensor, etc.). After receiving the responses from each device, the master control device can identify the corresponding device as an interactive device and record the corresponding response information.

[0095] S2. After device discovery is completed, the master control device can initiate a communication connection request for each interactive device, negotiate encryption algorithms and exchange key parameters with the interactive device, and complete two-way signature verification. Then, the master control device can establish a one-to-one independent encrypted communication channel with the interactive device. All subsequent transmissions of task instructions, interactive data, status information and event messages are transmitted through the corresponding encrypted channel to ensure that the data transmission is in a state of encryption protection throughout the entire process.

[0096] S3. After the encrypted communication channel is established, the master control device uses its own system clock as a reference or a unified external standard time source as a reference to send clock synchronization frames to each connected interactive device to calibrate the local system clock of each interactive device, so that the time reference of the master control device is consistent with that of all interactive devices, and ensures that the events generated by each device and the data collected have a unified timing reference standard.

[0097] This embodiment employs a three-step pre-processing approach: device pre-discovery, channel pre-encryption, and clock pre-synchronization. On one hand, it pre-determines the range of candidate devices suitable for cross-device migration, eliminating the time-consuming device scanning and connection establishment after scene switching triggers, effectively improving the response speed and smoothness of task migration. On the other hand, it constructs end-to-end encrypted channels to build full-link data security protection and eliminates timing deviations between multiple devices through a unified clock reference, ensuring the consistency of task status data and the reliability of the migration process from the bottom layer, providing a solid foundation for seamless and secure continuation of cross-device tasks.

[0098] Figure 5 This is a schematic diagram of the structure of a cross-device task migration system provided in an embodiment of the present invention. Figure 5 As shown, the system includes: The task initiation and status maintenance module 51 is used to initiate the target interaction task through the first interaction device in the first interaction scenario, and to maintain the execution status data of the target interaction task locally on the main control device. The migration triggering and permission verification module 52 is used to respond to the scene switching triggering event. If the target interaction task is in an incomplete state, the second interaction device in the second interaction scene is determined, and the second interaction device is subjected to succession permission verification. The task continuation module 53 is used to generate continuation interaction data based on the unfinished part of the target interaction task in the execution status data after the continuation permission verification is passed, and send the continuation interaction data to the second interaction device so as to continue the human-computer interaction process of the target interaction task through the second interaction device.

[0099] Furthermore, based on the above embodiments, the task initiation and status maintenance module 51 includes: The request receiving and verification unit is used to receive user interaction requests sent by the first interactive device and to verify the user interaction requests.

[0100] The task initiation unit is used to perform semantic parsing and intent recognition on the user interaction request after the user interaction request has been verified, so as to start the corresponding target interaction task.

[0101] The status maintenance unit is used to generate and continuously update the execution status data of the target interactive task locally on the main control device. The execution status data includes at least: task identifier, task execution progress, task progress snapshot, task parameters, and task context.

[0102] Furthermore, based on the above embodiments, the migration triggering and permission verification module 52 includes: The scene switching trigger unit is used to receive the departure event and task progress snapshot reported by the first interactive device, and use the departure event as the scene switching trigger event; wherein, the departure event is generated by the first interactive device when it detects that the user leaves its interactive area.

[0103] The task progress judgment unit is used to determine whether the target interactive task is in an incomplete state based on the task execution progress in the execution status data.

[0104] The task interruption unit is used to interrupt the interaction channel with the first interaction device and write a task progress snapshot to the execution status data if it is determined that the target interactive task is in an incomplete state.

[0105] The query instruction sending unit is used to send device status query instructions to each candidate interactive device that is communicatively connected to the main control device.

[0106] The status information receiving unit is used to receive the connection readiness status information sent by each candidate interactive device. The connection readiness status information is used to indicate whether each candidate interactive device has the conditions to continue the human-computer interaction process of the target interactive task.

[0107] The second interactive device determination unit is used to determine, based on the connection readiness status information, a device that matches the second interactive scenario and meets the connection conditions from among the candidate interactive devices as the second interactive device.

[0108] The continuity permission verification unit is used to perform continuity permission verification on the second interactive device. The continuity permission verification includes at least one of the following: device binding relationship verification, task ownership verification, device interaction permission verification, and migration compliance verification.

[0109] Furthermore, based on the above embodiments of the invention, the second interactive device determining unit is specifically used for: Based on the connection readiness status information, candidate interaction devices that match the second interaction scenario and meet the connection conditions are identified as available devices. If there is only one available device, then the available device is designated as the second interactive device. If there are multiple available devices, one of them will be selected as the second interaction device according to a preset selection strategy.

[0110] Furthermore, based on the above embodiments of the invention, the task succession module 53 includes: The available device determination unit is used to determine the candidate interaction devices that match the second interaction scenario and meet the connection conditions as available devices based on the connection readiness status information.

[0111] The first device determination unit is used to determine the available device as the second interactive device if the number of available devices is one.

[0112] The second device determination unit is used to determine one of the multiple available devices as the second interactive device according to a preset selection strategy if there are multiple available devices.

[0113] Furthermore, based on the above embodiments of the invention, the cross-device task migration system also includes: The migration termination confirmation module is used to terminate the migration process if no user feedback instruction is received or the user feedback instruction is a refusal to continue.

[0114] The device recovery instruction sending module is used to send a device recovery instruction to the first interactive device. The device recovery instruction is used to instruct the first interactive device to restore the interactive state corresponding to the target interactive task, so as to switch the interactive channel of the target interactive task back to the first interactive device.

[0115] The migration cancellation instruction sending module is used to send a migration cancellation instruction to the second interactive device. The migration cancellation instruction is used to instruct the second interactive device to release the temporary interactive resources associated with this task migration.

[0116] Furthermore, based on the above embodiments of the invention, the cross-device task migration system also includes an initialization module, used for: Before initiating the target interactive task, identify the interactive devices within the preset communication range; Establish encrypted communication channels with each interactive device separately; Each interactive device is synchronized with its own clock.

[0117] The cross-device task migration system provided in this embodiment of the invention can execute the cross-device task migration method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0118] To facilitate a better understanding of this technical solution by those skilled in the art, the following example illustrates the cross-device task migration method provided in this embodiment, using the example of a user initiating a restaurant search task at home via a large home screen and then continuing the task via AR glasses after leaving the home.

[0119] Figure 6 This is a timing diagram of a cross-device task migration method provided in an embodiment of the present invention. Figure 6 As shown, the core application scenario of this method is: when a user initiates a voice interaction to inquire about a dinner party plan through a large home screen, and leaves the home scene to go out, the interaction task is seamlessly transferred from the large home screen to the AR glasses worn by the user, so as to achieve uninterrupted continuation of the interaction scene and meet the user's need for continuous interaction from a fixed home scene to a mobile scene.

[0120] The functional modules and terminal devices involved in this embodiment are as follows. The core decision-making and data storage capabilities of all modules converge on the user's mobile phone, and the boundaries of rights and responsibilities are clear and non-overlapping: (1) Home large screen (edge): It only has basic input and output capabilities, equipped with a microphone array, RGB+TOF dual camera, and display module. It is responsible for audio acquisition, image acquisition, and basic UI rendering. It does not run intelligent agent programs, does not perform semantic understanding and decision processing, and is only used as a remote peripheral of the task decision module.

[0121] (2) AR glasses (edge): Only have light interaction capabilities, equipped with optical waveguide display module, bone conduction audio module, wear detection and eye tracking sensor, responsible for displaying prompts, voice acquisition, and basic status reporting. They do not store task context, do not perform semantic parsing and decision-making, and all temporary data are stored only in the running memory and are cleared immediately after disconnection.

[0122] (3) User Mobile Phone: As the physical carrier of this system, it houses the task decision module, device sensing gateway, task storage engine, user trusted database, semantic parsing module, and task scheduling engine. It is the physical foundation for the system's core computing power and trusted storage. The specific functional modules are as follows: ① Task Decision Module: This can refer to the unique task manager and decision core deployed locally on the user's mobile phone. It is responsible for the status management of the entire device, the entire process of task processing, migration decisions and data storage scheduling. It is the only trusted decision node in the entire system, and the core arbitration logic of all cross-device collaborative tasks is executed in this module.

[0123] ② Device awareness gateway: This can refer to a multi-device collaborative communication hub deployed locally on the user's mobile phone. It is responsible for the automatic discovery of interactive devices within the local area network, the establishment and maintenance of encrypted and trusted channels, the precise clock synchronization of all devices, the reporting and aggregation of device status information, and the forwarding and distribution of control commands. It is the essential communication hub for all data interaction between multiple devices.

[0124] ③ Task storage engine: This can refer to the data persistence storage module deployed locally on the user's mobile phone. It adopts a hybrid storage architecture that combines Redis hot storage and MySQL cold storage. It is responsible for the encrypted storage and read / write management of task identifiers, dialogue context data and task execution progress, ensuring that task data never leaves the local mobile phone.

[0125] ④ User Trusted Database: This can refer to an encrypted database module deployed in the local TEE trusted execution environment of the user's mobile phone. It is responsible for the secure and persistent storage of user device binding relationships, real-time status of each device, user operation permissions and interaction preference settings. It is the underlying source and verification basis for all trusted data in the system.

[0126] ⑤ Semantic parsing module: This can refer to the speech and semantic understanding module deployed locally on the user's mobile phone. It is responsible for core semantic parsing tasks such as Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU). All core logic of this module can be deployed in the local TEE environment of the mobile phone, and cloud AI services can be used for computing power acceleration. In the offline state, the local module can completely replace the cloud capabilities to ensure system availability.

[0127] ⑥ Task scheduling engine: This can refer to the task orchestration and execution management module deployed locally on the user's mobile phone. It is responsible for converting user intentions into executable directed acyclic graph task flows, managing calls to external service interfaces and timeout retries, monitoring the execution status of subtasks and sending back execution results. It is the execution exit point for the system to connect with third-party service capabilities.

[0128] Before executing the interactive task, the following multi-device trusted networking and pre-initialization process can also be included: S1. The device awareness gateway sends service discovery broadcasts to the home LAN via mDNS and DNS-SD protocols to detect all terminal devices that support these protocols. Upon receiving the broadcast, the home screen and AR glasses respectively send feedback of their device information to the device awareness gateway. Specifically, the information received by the home screen may include: a unique device identifier, hardware capability set (8-microphone array, RGB+TOF dual camera, 4K display), and firmware version number. The information received by the AR glasses may include: a unique device identifier, hardware capability set (waveguide display, bone conduction audio, eye tracking, multi-sensor fusion), and wear detection drive status, etc.

[0129] S2. After completing device discovery, the device sensing gateway initiates a TLS 1.3 (Transport Layer Security Protocol version 1.3) two-way authentication handshake with the home screen, task decision module, and AR glasses respectively, and exchanges the national cryptographic SM2 asymmetric encryption public key. After each terminal completes two-way signature verification, it establishes an end-to-end encrypted trusted channel with the device sensing gateway. All subsequent data interactions are transmitted through this encrypted channel to prevent data leakage and tampering.

[0130] S3 and the device sensing gateway send Precision Time Protocol (PTP) clock synchronization messages to the home screen, task decision module, and AR glasses, respectively, carrying the master clock timestamp; each terminal replies with the local clock offset and path delay measurement results; based on the measurement results, the device sensing gateway sends clock calibration parameters to each terminal to complete the microsecond-level clock synchronization of all devices, providing a foundation for subsequent session timing consistency and data frame alignment.

[0131] S4: The home screen preloads the Image Signal Processor (ISP) driver, microphone array audio driver, and TOF depth sensor driver locally, completing the data acquisition hardware readiness; the AR glasses preload the eye tracking driver, infrared proximity sensor driver, and bone conduction audio driver locally, completing the interaction hardware readiness.

[0132] S5 and the device perception gateway initiate cross-device encoding and decoding capability negotiation with the home screen, task decision module, and AR glasses respectively, and synchronize the unified parameters of OPUS audio encoding and H.264 video encoding; each terminal completes the alignment of encoding and decoding parameters, the interaction capability set negotiation is completed, and the negotiation results are sent back to the device perception gateway.

[0133] S6. The device sensing gateway encrypts and writes the online status of all devices, trusted channel status, clock synchronization status, and capability negotiation results into the user trusted database on the mobile phone. The user trusted database returns a status persistence success receipt to the device sensing gateway, and the entire link enters the ready-to-interact state.

[0134] The specific process of task initiation and end-side audio preprocessing includes: S1. The user issues a voice command to the large screen: "Check restaurants for weekend gatherings".

[0135] S2. The 8-microphone circular array of the home screen simultaneously acquires the voice signal. The user's sound source angle is estimated and locked through Direction of Arrival (DOA) estimation to filter out environmental interference. Then, the following operations are performed in sequence: ① Linear acoustic echo cancellation (AEC) combined with nonlinear residual echo suppression filters out the background music echo played by the home screen's own speakers; ② Ambient noise suppression (ANS) based on Wiener filtering and non-stationary noise suppression (NSNS) filters out the background noise of the home environment; ③ Automatic gain control (AGC) and voice activity detection (VAD) are performed to align the audio amplitude and extract the effective voice frames.

[0136] S3. The home screen extracts Mel-Frequency Cepstral Coefficients (MFCC) features from the captured valid speech frames, and then inputs them into a lightweight Deep Neural Network (DNN) wake-up model deployed on the device side, outputting a wake-up confidence score between 0 and 1. When this wake-up confidence score exceeds a preset threshold, the home screen determines that it has been legally woken up and switches its state to task active state.

[0137] S4. The home screen performs OPUS encoding on the pre-processed valid Pulse Code Modulation (PCM) audio frames, and then encapsulates the encoding result into Real-time Transport Protocol (RTP) data packets, marking the precise PTP timestamp and sequence number synchronized in the previous stage in the RTP packet header. The home screen reports the task activation event, user ID, device ID, and the encrypted RTP audio stream to the device awareness gateway through the encrypted trusted channel established in the previous stage.

[0138] S5. After receiving the encrypted RTP audio stream, the device awareness gateway first calls the task decision module to perform audio integrity verification within the TEE: It calculates a digest value for the audio payload using the State Cryptography Administration's hash digest algorithm, compares it with the digest value carried by the home screen in the packet, and confirms consistency, proving that the audio has not been tampered with during transmission; then it checks the continuity of the RTP sequence number to confirm no packet loss. After successful verification, the device awareness gateway forwards the audio stream to the semantic parsing module, completing the task initiation process.

[0139] The semantic parsing and session context creation process specifically includes: S1. After receiving the audio stream, the semantic parsing module performs end-to-end real-time speech recognition. Specifically, it performs audio framing and feature extraction on the audio stream, then inputs it into the end-to-end speech recognition model of the Transformer architecture for inference. At the same time, after combining hot word enhancement, punctuation restoration and spoken language smoothing, it finally outputs standardized recognized text: "Find restaurants for weekend gatherings".

[0140] S2. The dialogue management subunit of the semantic parsing module inputs the recognized text into the NLU subunit. The NLU subunit sequentially performs word segmentation, part-of-speech tagging, named entity recognition, and user referencing resolution. Through intent classification model reasoning, it outputs the core intent as "local life - restaurant query". Through slot filling model reasoning, it extracts filled slots: time = weekend, scenario = dinner party, and marks missing slots to be filled, such as the number of people at the dinner party.

[0141] S3, the semantic parsing module generates a globally unique task ID based on the snowflake algorithm, and associates it with the user ID and PTP timestamp to ensure the global uniqueness and traceability of the task ID; at the same time, it generates structured task context data, including user intent, slot information, dialogue history, and main interaction device ID, and performs embedding vector embedding on the dialogue text to generate semantic vectors for subsequent task continuation.

[0142] S4. The semantic parsing module initiates a write request to the task storage engine deployed locally on the phone, carrying the task ID, structured intent slot data (intent type, time slot value, list of slots to be filled), dialogue history (user questions and system response records for the current round), and semantic vector. The task storage engine performs an atomic operation (SET if Not eXists, SETNX) on the Redis database to write the data to Redis hot storage (with a 24-hour lifespan) and simultaneously writes it asynchronously to MySQL cold storage for persistence, recording a binary log (binlog) operation log. During the write process, the task storage engine uses the Redis Distributed Lock (Redlock) algorithm to add a distributed lock to the task ID, with a lock timeout of 30 seconds to prevent other devices from concurrently writing to the same task. After the write is complete, the engine sends a success confirmation to the semantic parsing module and returns a data lock token.

[0143] S5. The semantic parsing module writes the user ID, task ID, session permission token, and semantic vector index to the local user trusted database on the mobile phone. After the user trusted database completes data persistence, it returns a task creation completion receipt.

[0144] S6. The semantic parsing module sends a task creation success receipt and a "Querying..." UI rendering command to the task decision module. The task decision module forwards this command to the home screen via an encrypted channel through the device awareness gateway. The home screen then renders and displays the "Querying..." status UI, which could be a circular progress indicator ring with a semi-transparent text label. Next, the home screen reports the UI rendering completion status to the device awareness gateway, which synchronously writes this status into the user's trusted database, completing the entire task initialization process.

[0145] The specific process for user departure detection and session interruption handling includes: S1. The user leaves the home screen, triggering the prerequisite for this task migration. The RGB color camera and TOF depth camera of the home screen simultaneously acquire images and depth point cloud data at a frame rate of 30fps; the ISP module performs automatic exposure, automatic white balance, image noise reduction, and distortion correction to complete image preprocessing; human keypoint detection and face tracking are performed through edge-side YOLO+OpenPose model inference; face trajectory is tracked through kernelized correlation filters (KCF) filtering algorithm. When no valid face / human keypoints are detected for several consecutive frames (e.g., 10), the user's presence confidence is calculated. When the confidence is lower than a preset threshold, the user's departure is triggered; the home screen switches its state from task active state to interrupted and waiting state, while saving a snapshot of the current task progress and releasing unnecessary hardware resources. This snapshot may include the following information: filled slot data (time = weekend), a list of slots to be filled (number of people, location), dialogue history (a complete voice sentence entered by the user and the semantic results processed by the system), current task stage identifier (slot filling in progress - number of people waiting for users to add), and PTP timestamp of the user's last interaction.

[0146] S2. The home screen reports an absence event to the device awareness gateway via an encrypted channel, along with the task ID, the PTP interruption timestamp at the time of absence, and a task progress snapshot. Upon receiving this information, the device awareness gateway immediately updates the corresponding user's presence status field in the user trusted database to "left the home screen" and marks the home screen's device status field as "interrupted and awaiting migration." The user trusted database returns a status update success receipt.

[0147] S3. The device awareness gateway forwards the departure event, task ID, and task progress snapshot to the semantic parsing module, triggering the session interruption handling process. Upon receiving the departure event, the semantic parsing module initiates a task interruption update request to the task storage engine, carrying the task ID and the lock token obtained in the previous stage (this token is stored in the phone's local memory). After verifying the lock token's validity, the task storage engine performs a database update operation, changing the task status field to "interrupted and awaiting migration," changing the task lifecycle field from "active" to "suspended," writing the interruption node information (interruption time, interruption reason), and simultaneously writing the task progress snapshot to the snapshot field of the execution status data. After the operation is complete, the task storage engine re-enters the Redlock distributed lock, ensuring that concurrent writes to the current task are strictly prohibited from being performed by the home screen or other devices before the migration is complete.

[0148] S4. The device perception gateway sends a device status query command to the AR glasses to verify the device's online status, wearing status, and hardware readiness status. The AR glasses perform multi-sensor fusion sampling through infrared proximity sensors, capacitive sensors, and a 6-axis gyroscope. Using a wearing status model, they calculate the wearing confidence level. If the confidence level exceeds a preset threshold, it confirms that the user is wearing the glasses. The AR glasses then pre-wake up the eye-tracking module, bone conduction audio module, and waveguide display module, complete hardware initialization, switch their state to "ready to migrate," and send a response to the device perception gateway containing triple status information: online, worn, and hardware ready, as well as device capability sets and wearing confidence level data.

[0149] S5. The device sensing gateway updates the AR glasses status to "ready to migrate" in the user trusted database, indicating that the user's current interactive device is the AR glasses; the user trusted database returns a status update success receipt.

[0150] S6. The device awareness gateway synchronizes data such as departure events, AR glasses readiness status, task interruption snapshots, and task IDs to the task decision module. The task decision module performs the following four-fold permission verification within the TEE: device binding relationship verification, task ownership verification, device interaction permission verification, and migration compliance verification. After all four verifications pass, the task decision module generates a unique migration token based on the national cryptographic algorithm SM2. This token contains fields such as task ID, target device ID, permission scope (for single use only in this task), and validity period (e.g., valid for 60 seconds), and digitally signs the token using the task decision module's private key. The task decision module then distributes this token to the AR glasses through the device awareness gateway, simultaneously marking it with a PTP timestamp as proof for subsequent confirmation interactions.

[0151] The process of migration prompts and trusted user confirmation interactions specifically includes: S1. The task decision module sends a task migration prompt instruction to the AR glasses through the device perception gateway. This instruction includes: migration token (SM2 signed data packet), task ID, and prompt text. S2. Upon receiving the instruction, the AR glasses perform the following verifications: ① Verify the SM2 signature of the migration token using the public key of the task decision module to confirm that the token was indeed issued by a legitimate task decision module and has not been tampered with; ② Verify the task ID attribution to confirm that the user account corresponding to the task ID matches the user credentials not stored locally; ③ Verify the validity of the PTP timestamp to confirm that the deviation between the token timestamp and the current local time of the AR glasses is within a preset threshold to prevent replay attacks. After all verifications pass, the AR glasses switch their state to "migration pending confirmation state".

[0152] S3: The AR glasses send a voice synthesis request for the prompt text "Continue searching for restaurants?" to the semantic parsing module (which can optionally use cloud-based TTS services for computational acceleration). This requires adaptation to the bone conduction speaker parameters. The semantic parsing module returns an OPUS-encoded audio file. The AR glasses complete the audio decoding, perform bone conduction-specific EQ equalization adjustments, and simultaneously perform UI distortion correction and field-of-view adaptation through the waveguide display module, rendering the prompt bubble UI. The waveguide prompt bubble is displayed to the user, while a soft prompt tone is played via bone conduction: "Continue searching for restaurants?".

[0153] S4. The AR glasses report a status event to the device perception gateway: the prompt has been displayed and is awaiting user confirmation. The device perception gateway synchronously writes this status to the user trusted database, updates the task status to "migration awaiting user confirmation," and the synchronization of the status across all devices is completed. The user trusted database returns a status update success receipt.

[0154] S5. Users can respond in two ways: eye-tracking confirmation (eye-tracking lock on the confirmation area ≥ 500ms) or voice reply "continue". If it is eye-tracking confirmation, the AR glasses use an infrared eye-tracking camera to capture pupil images, perform pupil localization and gaze point calculation, and use on-device model inference to calculate the duration of gaze in the fixation area. Confirmation is completed when the confidence level of the confirmation command is greater than a preset threshold. If it is voice confirmation, the AR glasses use a bone conduction microphone to capture vibration audio, perform bone conduction audio enhancement, noise reduction, and VAD detection, and use on-device offline small-vocabulary ASR inference to match confirmation keywords to complete the confirmation.

[0155] S6. After confirmation, the AR glasses generate an encrypted confirmation signal, which includes the following fields: confirmation method (e.g., gaze confirmation), PTP timestamp, task ID, and device ID. The AR glasses then execute the following steps on this signal: ① Generate a 256-bit digest value using the national cryptographic SM3 hash digest algorithm; ② Digitally sign the digest value using the AR glasses' own SM2 private key. Finally, the AR glasses package the encrypted confirmation signal, the signed data, and the task receipt, and upload them to the device perception gateway via an encrypted channel.

[0156] S7 and the device awareness gateway synchronize the user confirmation event, confirmation signal, and signature data to the task decision module and semantic parsing module, respectively. The task decision module performs confirmation signal verification within the TEE environment: ① Decrypt the signature data using the AR glasses' public key to obtain digest value A; ② Perform SM3 hash calculation on the received confirmation signal plaintext to obtain digest value B; ③ Compare digest value A and digest value B; if they match, the confirmation signal was indeed issued by the AR glasses and has not been tampered with; ④ Verify that the deviation between the PTP timestamp and the current mobile phone clock is within a preset threshold, confirming it is not a replay attack. After all verifications pass, the task decision module returns a "Confirmation signal verification passed, task migration effective" receipt to the semantic parsing module.

[0157] S8. After receiving the signature verification confirmation, the semantic parsing module sends a task unlocking request to the task storage engine, carrying the task ID, lock token, and confirmation signal. The task storage engine performs the following operations: ① Verifies the validity of the lock token, confirming that the requesting party has the right to operate the task; ② Releases the Redlock distributed lock, releasing the exclusive lock on the task; ③ Performs a database UPDATE operation: updates the main interaction device field to "AR Glasses", the task status field to "Migration in Progress", and the task lifecycle field to "Reactivated"; ④ Writes all migration link records to the migration log table, including departure time, migration trigger time, confirmation time, source device ID, target device ID, network latency, etc. After the operation is completed, it returns a successful unlocking confirmation, full task context data, and semantic vector to the semantic parsing module.

[0158] S9. The semantic parsing module updates the user's primary interaction device to AR glasses in the user's trusted database, and the home screen's status is "session suspended in the background"; the user's trusted database returns a status update success receipt, and the transfer of primary device permissions is complete. The device perception gateway sends a task suspension command to the home screen. After receiving the command, the home screen releases hardware resources and enters a low-power suspension state.

[0159] The specific process of task context continuation and task scheduling execution includes: S1. The task decision module initiates a context retrieval request to the task storage engine, carrying the task ID, user permission token, and master device handover credential (migration token). After the task storage engine verifies all credentials, it returns a full task context data packet, including: core intent (local life - restaurant query), filled slots (time = weekend), list of slots to be filled (number of people, location), dialogue history (complete dialogue record of the current round), interruption progress snapshot, and semantic vector.

[0160] S2. The task decision module performs format conversion and data anonymization on the above data to adapt it to the AR glasses' edge model inference specifications. After adaptation, the task decision module synchronizes the following data to the AR glasses through an encrypted channel via the device perception gateway: task ID, anonymized full context, dialogue continuation instructions, and semantic vectors, completing cross-device context migration.

[0161] The S3 and AR glasses encrypt and cache the aforementioned data within the local TEE environment (stored in running memory, cleared upon power failure), and switch their status from "migration pending confirmation" to "task active state," locking the local bone conduction microphone and waveguide display hardware resources to ensure exclusive use for this task. They then report to the device perception gateway that context synchronization is complete and task continuation is successful.

[0162] S4. The dialogue management subunit of the task decision module performs dialogue strategy reasoning based on the current task context. Specifically, the dialogue management subunit knows the following information: the intent is a restaurant query, the time is determined to be a weekend, the user's city is Beijing (read from the user's trusted database), and the current missing slot is "number of people". Then, it uses a natural language generation model to reason and generate the following dialogue text: "Approximately how many people are there?".

[0163] The text is synthesized into speech via the TTS submodule of the semantic parsing module, with synthesis parameters adapted to the characteristics of the AR glasses' bone conduction speakers. The task decision module then sends the continuation dialogue text, synthesized audio, and rendering parameters to the AR glasses via an encrypted channel through the device perception gateway.

[0164] The S5 AR glasses' waveguide module performs UI adaptation rendering, while the audio module performs decoding and EQ adjustment. The AR glasses then display a text bubble prompting the user, "Approximately how many people are there?" and play the corresponding audio via bone conduction to guide the user in providing necessary information.

[0165] S6. Write the guide dialogue to the task history in the session storage engine and update the task status to "slot filling"; the session storage engine returns a successful write receipt, and the task lifecycle proceeds normally.

[0166] S7. The user replies to the AR glasses via voice: "4 people". The AR glasses collect vibration audio signals through the bone conduction microphone, and perform bone conduction audio enhancement, noise reduction, and echo cancellation; they capture valid voice frames through VAD voice activity detection, perform OPUS encoding, and generate RTP data packets; they upload the encrypted user reply audio stream, task ID, and PTP timestamp to the device perception gateway.

[0167] S8 and the device awareness gateway synchronously forward the user's response audio stream and metadata to the task decision module and semantic parsing module, respectively. The ASR submodule of the semantic parsing module identifies the audio as the text "4 people," and the NLU slot filling model fills this value into the "number of people" slot, marking it as filled. The dialogue management submodule checks all required slots: the slots required for the intent to query the restaurant include time, number of people, and location (the location has been read from the user's trusted database, with the user's current city being Beijing as the default). After confirming that all required slots have been filled, it triggers downstream task scheduling.

[0168] S9. The semantic parsing module sends a write request to the task storage engine to write the user's reply dialogue history, complete the slot data (number of people = 4, location = Beijing), and update the task status to "pending execution". The task storage engine returns a success receipt.

[0169] S10. The semantic parsing module initiates a restaurant query subtask creation request to the task scheduling engine, associating the main task ID, query parameters (time = weekend, number of people = 4, location = Beijing, scenario = group dining), and permission token. The task scheduling engine generates a unique subtask ID, constructs a Directed Acyclic Graph (DAG), sets task priority, timeout, and retry strategy; allocates task execution resources, calls the third-party Point of Interest (POI) service interface, performs request signing and parameter encapsulation; initiates interface requests, monitors task execution status in real time, handles timeout retries and exception fallback; and returns the subtask execution status and progress receipt to the semantic parsing module.

[0170] S11. The semantic parsing module sends the task processing status command and progress parameters (such as the current step: "Restaurant search completed, 5 restaurants found") to the task decision module. The task decision module forwards this status command to the AR glasses through the device perception gateway. The AR glasses render the progress UI "Searching for restaurants that meet your requirements..." through the optical waveguide, completing the task scheduling and execution process.

[0171] S12. The Natural Language Generation (NLG) submodule of the Semantic Parsing module generates recommended text and multi-turn dialogue guidance scripts (for subsequent interaction guidance) based on the query results. The TTS submodule of the Semantic Parsing module synthesizes the above text into audio adapted for bone conduction in AR glasses. The Semantic Parsing module sends the query result text, synthesized audio, and AR spatial rendering parameters to the Task Decision module. The Task Decision module sends the above data to the AR glasses through the device perception gateway.

[0172] S13. After receiving the above data, the AR glasses perform POI data spatial coordinate transformation and 3D rendering adaptation, display the restaurant search results to the user, and simultaneously play recommended voice messages via bone conduction. S14. The semantic parsing module sends a write request to the task storage engine, updates the task status to "processing completed", updates the task lifecycle to "pending closure", and writes the final result data (such as the queried restaurant data). The task storage engine returns a successful update receipt.

[0173] S15 and the device sensing gateway synchronize the task completion status to the home screen, the task decision module, and the AR glasses, respectively. After synchronization, all devices in the entire system reach a consensus on the execution status of this task. At the same time, it supports the main device to switch back to the home screen at any time.

[0174] S16. If there are no new interactions within the preset time period (e.g., 24 hours) after this task, the task storage engine will trigger an automatic archiving process: Redis hot data will be deleted upon expiration, MySQL cold data will be encrypted and archived, and object storage will back up the entire session log. At this point, the task completes its full lifecycle loop.

[0175] In one embodiment, the following fallback rollback process is also included: S1. The user responds to the migration prompt by shaking their head or saying "cancel". At this time, the AR glasses report the user's refusal to continue the task signal and signature data to the device perception gateway. The device perception gateway synchronizes the user's refusal event to both the task decision module and the semantic parsing module.

[0176] S2. Upon receiving a user rejection event, the semantic parsing module initiates a task rollback request to the task storage engine, carrying the task ID and lock token. After successful verification, the task storage engine releases the Redlock distributed lock and executes a database transaction rollback: updating the task status field from "Interrupted and Pending Migration" to "Interrupted and Terminated," and switching the main interactive device back to the home screen. A rollback success receipt is then returned to the semantic parsing module.

[0177] S3. The semantic parsing module initiates a status update request to the user-trusted database, updating the task status to "interrupted recovery," with the main interaction device being the home screen. The user-trusted database returns a successful update receipt.

[0178] S4. The device sensing gateway sends a task recovery command to the home screen, which wakes up the hardware resources and switches back to the session active state; it also sends a migration cancellation command to the AR glasses, which releases the resources and switches back to the idle standby state.

[0179] Thus, the entire system completed a full transaction rollback in the abnormal scenario where the user refused to migrate. The home screen returned to its interactive state before the interruption, and the AR glasses released all temporary resources without generating any dirty data or losing any tasks, achieving reliable recovery from the abnormal scenario.

[0180] This embodiment achieves seamless task migration between a home screen (for display only) and AR glasses (for lightweight interaction) through a clear responsibility architecture with the task decision module as the sole decision-making center. All task context and status data are stored locally on the mobile phone within a trusted TEE environment, without relying on real-time cloud transmission, allowing migration to be completed even when offline. By combining YOLO+OpenPose visual departure detection on the home screen with multi-sensor (infrared proximity + capacitive + gyroscope) wear detection on the AR glasses, automated migration triggering without user intervention is achieved. Before data handover, multi-dimensional verification of device binding relationships, task ownership, interaction permissions, and migration compliance is performed, coupled with TLS 1.3 two-way authentication, national cryptographic SM2 / SM3 encryption, and digital signature systems, ensuring the security and trustworthiness of the migration process. Through distributed locks (Redlock) and a complete transaction rollback mechanism, the system state can be restored without loss in abnormal scenarios such as user rejection, demonstrating strong engineering robustness. The entire process provides users with a continuous, secure, and seamless interactive experience across fixed home scenarios and mobile outing scenarios.

[0181] Figure 7 This is a schematic diagram of a multi-terminal interactive system based on a multi-agent collaborative architecture, provided as an embodiment of the present invention. Figure 7 As shown, the system includes: (1) Personal Claw 61: Deployed on the user's mobile terminal and AR glasses, it is used to handle personal affairs and interact with the user in a personalized way. Personal Claw consists of two carriers: AR glasses and mobile terminal. The AR glasses serve as a quick entry point and are equipped with an optical waveguide display module, microphone array (voice acquisition), camera (gesture recognition, environmental perception), inertial measurement unit (IMU, head posture tracking), infrared sensor, and 5G / Wi-Fi / Bluetooth Low Energy (BLE) communication module, supporting independent communication of the embedded Subscriber Identity Module (eSIM). The mobile terminal serves as the local data carrier and general entry point of Personal Claw, running a lightweight artificial intelligence model to achieve offline voice wake-up, local task execution, and data caching. The privacy data of Personal Claw (such as personal schedules and communication records) is stored only locally on the user's mobile phone and is not uploaded to the cloud.

[0182] (2) Family Claw 62: Deployed at a shared home terminal—in the form of a large, physical display screen that fits against a wall (such as a seamless MicroLED splicing screen or an ultra-short-throw projector + diffuse reflection hard screen) for managing family members' public affairs and multi-person interactions within the family. Its hardware includes: a physical display module (using opaque display technology to give the virtual butler image a "materialized" visual effect, customizable as a real person or an animated virtual image), a multimodal sensing array (wide-angle cameras for face recognition and gaze recognition, a microphone array for voiceprint separation and beamforming, and a depth sensor for gesture detection and spatial touch perception), a local edge computing unit (running the core logic of the family claw, supporting offline operation), and a communication module (connected to the home broadband in real time, communicating and controlling with a cloud-based collaborative server and other smart devices within the home). The Family Claw has a persona and emotional state independent of any individual family member.

[0183] (3) Friend Claw 63: Primarily deployed in the cloud, while retaining lightweight copies in the user's AR glasses and mobile application (APP), it is used to maintain the interaction and relationship management between the user and members of the social network. Friend Claw takes on the proactive and independent role of proactive social interaction, proactive consultation, and relationship maintenance. It can proactively analyze the user's social graph (local analysis, without uploading to the cloud), generate human-like social maintenance suggestions, and present them in a natural way through AR glasses.

[0184] (4) Cloud Collaboration Server 64: As the system's central hub, it includes a task synchronization module (managing the creation, modification, confirmation, and state synchronization of tasks among various agents), a session state management module (maintaining the task context between users and agents, supporting seamless task migration between agents), an access control module (defining user roles, task tags, and data access permissions), an active interaction decision module (generating active interaction events based on time, user status, environment, and historical interactions), and a persona and emotion engine (storing and updating persona parameters and emotional states of each agent to achieve cross-agent personality unification). The cloud collaboration server can also provide the following services: ① Spatial Anchoring: providing AR glasses with the ability to spatially align virtual content with the real environment; ② Visual Impairment Services: providing real-time voice navigation based on spatial detection by AR sensors.

[0185] This embodiment utilizes three intelligent agents—Personal Claw, Family Claw, and Friend Claw—to achieve data interoperability and state sharing via a cloud-based collaborative server. This constructs a unified collaborative system for personal affairs, family public affairs, and social relationship maintenance, effectively breaking the limitations of isolated intelligent agents in traditional solutions. Family Claw possesses a personality setting and emotional state independent of any family member, enabling differentiated information presentation and permission control based on task tags and user roles. This fundamentally solves the problems of command conflicts and confusion between public and private affairs in multi-user shared scenarios. Simultaneously, with the help of a conversation state management module and a personality and emotion engine, user interactions with any intelligent agent can seamlessly migrate across different terminal devices such as the family screen and AR glasses, maintaining personality consistency and ensuring the continuity of the interactive experience and the unity of the "virtual life." Friend Claw proactively generates maintenance suggestions based on the social graph and collaborates with Personal Claw to complete scheduling and multi-party coordination, upgrading social functions from passive recommendations to proactive, anthropomorphic relationship maintenance. In addition, the centralized management architecture of the cloud-based collaborative server gives the system strong horizontal scalability, enabling easy access to new intelligent agents such as CarClaw and RingClaw, thus providing an open and unified technical foundation for the future evolution of the multi-agent ecosystem.

[0186] Figure 8A schematic diagram of a master control device 70, which can be used to implement embodiments of the present invention, is shown. The master control device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The master control device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0187] like Figure 8 As shown, the main control device 70 includes at least one processor 71 and a memory, such as a read-only memory (ROM) 72 and a random access memory (RAM) 73, communicatively connected to the at least one processor 71. The memory stores computer programs executable by the at least one processor. The processor 71 can perform various appropriate actions and processes based on the computer program stored in the ROM 72 or loaded into the RAM 73 from storage unit 78. The RAM 73 can also store various programs and data required for the operation of the main control device 70. The processor 71, ROM 72, and RAM 73 are interconnected via a bus 74. An input / output (I / O) interface 75 is also connected to the bus 74.

[0188] Multiple components in the main control device 70 are connected to the I / O interface 75, including: an input unit 76, such as a keyboard, mouse, etc.; an output unit 77, such as various types of displays, speakers, etc.; a storage unit 78, such as a disk, optical disk, etc.; and a communication unit 79, such as a network card, modem, wireless transceiver, etc. The communication unit 79 allows the main control device 70 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0189] Processor 71 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 71 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 71 performs the various methods and processes described above, such as cross-device task migration methods.

[0190] In some embodiments, the cross-device task migration method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 78. In some embodiments, part or all of the computer program may be loaded and / or installed on host device 70 via ROM 72 and / or communication unit 79. When the computer program is loaded into RAM 73 and executed by processor 71, one or more steps of the cross-device task migration method described above may be performed. Alternatively, in other embodiments, processor 71 may be configured to perform the cross-device task migration method by any other suitable means (e.g., by means of firmware).

[0191] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0192] In some embodiments, the cross-device task migration method may be implemented as a computer program, which is implicitly included in a computer program product. When executed by a processor, the computer program implements the cross-device task migration method of the present invention. The computer program product can be understood as a software product that primarily implements its solution through a computer program. The computer program used to implement the method of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer program causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a standalone software package, or entirely on a remote machine or server.

[0193] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0194] To provide interaction with the user, the systems and techniques described herein can be implemented on a master control device, which includes: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the master control device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0195] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0196] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0197] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0198] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for cross-device task migration, characterized in that, Applied to a main control device, the method includes: The target interaction task is initiated through the first interaction device in the first interaction scenario, and the execution status data of the target interaction task is maintained locally on the main control device. In response to a scene switching event, if the target interaction task is in an incomplete state, the second interaction device in the second interaction scenario is determined, and the connection permission is verified for the second interaction device. After the succession permission verification is passed, succession interaction data is generated based on the incomplete part of the target interaction task in the execution status data, and the succession interaction data is sent to the second interaction device to continue the human-computer interaction process of the target interaction task through the second interaction device.

2. The method according to claim 1, characterized in that, The step of initiating the target interaction task through the first interaction device in the first interaction scenario and maintaining the execution status data of the target interaction task locally on the main control device includes: Receive user interaction requests sent by the first interactive device and verify the user interaction requests; After the user interaction request passes the verification, the user interaction request is semantically parsed and the intent is identified in order to start the corresponding target interaction task; The execution status data of the target interactive task is generated locally on the main control device and continuously updated and maintained; the execution status data includes at least: task identifier, task execution progress, task progress snapshot, task parameters, and task context.

3. The method according to claim 1, characterized in that, In response to a scene switching trigger event, if the target interaction task is in an incomplete state, the second interaction device in the second interaction scenario is determined, and the connection permission verification for the second interaction device is performed, including: The system receives an exit event and a task progress snapshot reported by the first interactive device, and uses the exit event as the scene switching trigger event; wherein, the exit event is generated by the first interactive device when it detects that the user has left its interactive area; Based on the task execution progress in the execution status data, determine whether the target interactive task is in an incomplete state; If the target interactive task is determined to be in an incomplete state, the interaction channel with the first interactive device is interrupted, and the task progress snapshot is written into the execution status data. Send device status query commands to each candidate interactive device that is communicatively connected to the main control device; Receive connection readiness status information sent by each of the candidate interaction devices, wherein the connection readiness status information is used to indicate whether each of the candidate interaction devices has the conditions to continue the human-computer interaction process of the target interaction task. Based on the connection readiness status information, a device that matches the second interaction scenario and meets the connection conditions is determined from the candidate interaction devices as the second interaction device. Perform a connection permission verification on the second interactive device. The connection permission verification includes at least one of the following: device binding relationship verification, task ownership verification, device interaction permission verification, and migration compliance verification.

4. The method according to claim 3, characterized in that, The step of determining, based on the connection readiness status information, a device from the candidate interaction devices that matches the second interaction scenario and meets the connection conditions as the second interaction device includes: Based on the connection ready status information, candidate interaction devices that match the second interaction scenario and meet the connection conditions are determined as available devices; If the number of available devices is one, then the available device is identified as the second interactive device; If there are multiple available devices, one of them is selected as the second interactive device according to a preset selection strategy.

5. The method according to claim 1, characterized in that, After the continuation permission verification is passed, continuation interaction data is generated based on the incomplete portion of the target interaction task in the execution status data, and the continuation interaction data is sent to the second interaction device to continue the human-computer interaction process of the target interaction task through the second interaction device, including: After the connection permission verification is passed, a connection confirmation prompt instruction is sent to the second interactive device, and a user feedback instruction returned by the second interactive device is received; If the user feedback instruction is to confirm the continuation, then task continuation breakpoint data is generated based on the task identifier, task execution progress and task progress snapshot in the execution status data, and interactive prompt information is generated based on the task parameters and task context in the execution status data. The task continuation breakpoint data and the interaction prompt information are used to generate the continuation interaction data; The interaction channel of the target interaction task is switched to the second interaction device, and the follow-up interaction data is sent to the second interaction device so that the second interaction device can continue the human-computer interaction process of the target interaction task.

6. The method according to claim 5, characterized in that, Also includes: If the user feedback instruction is not received or the user feedback instruction is a refusal to continue, the current task migration process will be terminated. Send a device recovery command to the first interactive device. The device recovery command is used to instruct the first interactive device to restore the interactive state corresponding to the target interactive task, so as to switch the interactive channel of the target interactive task back to the first interactive device. Send a migration cancellation command to the second interactive device, the migration cancellation command being used to instruct the second interactive device to release the temporary interactive resources associated with this task migration.

7. The method according to claim 1, characterized in that, Before initiating the target interactive task, the following is also included: Identify interactive devices within the preset communication range; Establish encrypted communication channels with each of the aforementioned interactive devices; Each of the aforementioned interactive devices is synchronized with its clock.

8. A cross-device task migration system, characterized in that, Deployed on the main control device, the system includes: The task initiation and status maintenance module is used to initiate a target interactive task through a first interactive device in a first interactive scenario, and to maintain the execution status data of the target interactive task locally on the main control device. The migration triggering and permission verification module is used to respond to scene switching triggering events. If the target interaction task is in an incomplete state, the module determines the second interaction device in the second interaction scenario and performs a succession permission verification on the second interaction device. The task continuation module is used to generate continuation interaction data based on the unfinished part of the target interaction task in the execution status data after the continuation permission verification is passed, and send the continuation interaction data to the second interaction device so as to continue the human-computer interaction process of the target interaction task through the second interaction device.

9. A master control device, characterized in that, The main control device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the cross-device task migration method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the cross-device task migration method of any one of claims 1-7.

Images