An interaction method, system, medium and device for a multi-user accompanying scene

Abstract

The application discloses a multi-user companion scene-oriented interaction method, system, medium and device. The method comprises the following steps: receiving a voice input stream; performing voice activity detection and automatic speech recognition to obtain text information; when the text information contains a wake-up word, entering a wake-up state and obtaining identity information associated with a current user; determining the current user identity based on the identity information according to predetermined identity determination conditions; if the determination is successful, locking the current active period to the corresponding registered user identity; if the determination fails, using a default isolated identity; keeping the current user identity unchanged within the current active period, and performing conversation management, memory injection and tool calling; when a farewell word or an idle timeout event is detected, ending the current active period, releasing the current user identity, and submitting a conversation snapshot for asynchronous long-term memory extraction. The application takes into account the identity recognition robustness, user privacy security and interaction continuity.

Description

Technical Field

[0001] This invention relates to the fields of human-computer interaction, speech recognition, and artificial intelligence, and in particular to an interaction method, system, computer-readable storage medium, voiceprint identification device, hierarchical memory management device, and asynchronous memory retrieval device for multi-user companionship scenarios. Background Technology

[0002] Home companionship, elderly care, and personal assistant devices are evolving from single-turn question-and-answer terminals into interactive intelligent agents capable of long-term coexistence and continuous dialogue. These devices are often placed in public spaces such as homes and nursing homes, shared by multiple people including the elderly, their children, and caregivers. Shared devices bring three core, interconnected problems: First, the issue of identity recognition and locking: the current speaker's identity may change as family members take turns using the device, but frequent re-identification during continuous dialogue can cause instability, leading to interruptions or confusion in personalized services. Second, the issue of privacy and memory isolation: users' long-term preferences, care backgrounds, reminders, and historical tasks are highly private; incorrect or missing identity determination can lead to misreading, mismodification, or misrepresentation of private information across users. Third, the issue of context and concurrency management: long-term companionship requires cross-conversation memory, but the context length of large language models is limited. Directly injecting the original history and all long-term memory increases latency and dilutes the current task. Simultaneously, concurrent activities such as voice interaction, tool calls, reminder broadcasts, and user interruptions require efficient coordination to avoid conflicts.

[0003] Existing technologies, such as EP1941411B1 and US10452816B2, while disclosing the basic path for identifying users and retrieving personalized profiles through voice, face, and other methods, primarily focus on the identification methods themselves. They fail to provide a complete closed-loop state machine from wake-up, identity locking, continuous dialogue, memory management to hibernation release, and especially lack engineering mechanisms to maintain identity stability and prevent cross-user memory misuse when handling multi-user switching. Other existing technologies, such as MemoryBank and MemGPT, propose hierarchical management schemes for long-term memory, but mainly address the model context window limitation problem and do not provide a unified design for identity locking, memory directory access permissions, user isolation, and asynchronous concurrency control for shared device scenarios.

[0004] Therefore, there is an urgent need for a technical solution that can solve the above problems, so as to realize the closed-loop operation of shared devices between low-interference sleep, reliable wake-up, stable identity, continuous dialogue, interruptible broadcast, asynchronous reminder and sleep again, and realize the extraction, classification and hierarchical injection of long-term memory isolated by individuals in this closed loop. Summary of the Invention

[0005] This invention aims to solve the technical problem in existing technologies where stability, privacy, and efficiency are difficult to balance in shared devices designed for multi-user companionship scenarios due to the dynamic coupling of identity, memory, and interaction. Specifically, this invention achieves this by solving the following technical problems: First, how to accurately bind user identity during the wake-up phase and maintain stability throughout the entire active period to avoid identity fluctuations; Second, how to achieve strict isolation of user memories to prevent privacy leaks, while ensuring that authorized users can access their personal memories; Third, how to efficiently manage context, concurrent tasks, and asynchronous operations to improve interaction fluency and response speed without interfering with front-end interaction.

[0006] The technical problem of this invention is solved by the following technical solution: In a first aspect, the present invention provides an interaction method for multi-user companionship scenarios, executed by an interaction system, comprising the following steps: S1, Receive voice input stream; S2. Perform voice activity detection and automatic speech recognition on the voice input stream to obtain text information; S3. When the text information contains a wake word, enter the wake-up state and obtain the identity identification information associated with the current user; S4. Based on the identity information, determine the current user's identity according to the predetermined identity determination conditions; if the determination is successful, lock the current active period to the corresponding registered user identity, and refuse to execute any new identity re-determination request within the current active period until the current active period ends; if the determination fails, use the default isolated identity, which has no right to access any registered user's long-term memory and personalized data. S5. Perform dialogue management, memory injection, and tool invocation based on the current user's identity; S6. When a farewell message or idle timeout event is detected, the current active period ends, the current user identity is released, and a session snapshot is submitted for asynchronous long-term memory extraction. The identity information includes biometric information and / or digital credential information; the predetermined identity determination conditions include: determining whether the comprehensive matching degree between the identity information and the registered user sample meets the identity locking conditions.

[0007] In some embodiments, the following technical features are also included: The identity identification information is a voiceprint feature; step S4 involves setting the predetermined identity determination conditions, specifically including the following steps: S41. Calculate the cosine similarity between the voiceprint embedding vector of the wake-up voice and the voiceprint embedding vectors of each registered user's voiceprint sample, and form a similarity matrix; S42. For each registered user, take the maximum similarity among all their voiceprint samples as the score of that registered user. S43. Identify the best candidate user with the highest score and the second-highest candidate user with the second-highest score; S44. Determine whether the score of the best candidate user is greater than or equal to the first threshold, and whether the difference between the score of the best candidate user and the score of the second candidate user is greater than or equal to the second threshold. S45. If so, then the identity of the best candidate user is locked. S46. If not, then the default isolation identity will be used.

[0008] In some embodiments, the values ​​of the first threshold and the second threshold can be dynamically adjusted according to the ambient noise level or the privacy mode set by the interactive system. The first threshold ranges from 0.65 to 0.75, and the second threshold ranges from 0.15 to 0.20.

[0009] In some embodiments, the identity information is facial features; the registered user samples are a facial image sample library.

[0010] In some embodiments, in step S5, the injected memory is layered, specifically including: The current round of user requests and assistant responses; The original dialogue history of several recent rounds; Compressed summaries of earlier rounds of dialogue; The semantic state of the current session, including open items, resolved items, and timeline; The current user's long-term memory index; Specific topic details specified in the long-memory index are injected on demand, only when relevant to the current request.

[0011] In some embodiments, no long-term memory indexes and topic details of registered users are injected for the current user using the default isolated identity.

[0012] In some embodiments, the tool invocation in step S5 includes the following steps: S51. Parse the tool call request output by the large language model; S52. Perform parameter specification and permission verification on the request; S53. Classify and handle errors according to their types: For non-retryable errors such as missing parameters or insufficient permissions, directly return the error code and a user-understandable message. S54. For retryable errors such as network timeout or service unavailability, return a retryable flag; S55. If the large language model does not provide an explanation for the tool call failure, the interactive system will directly broadcast a predefined, user-understandable message corresponding to the error code. The asynchronous long-term memory extraction described in step S6 includes the following steps: S61. Receive the submitted session snapshot; S62. Based on predefined filtering rules, filter out candidate content that should not be saved as long-term memory; the filtering rules include at least: filtering content that the subject user is not the current user and is not their authorized care object, filtering content containing sensitive identifiers or original biometrics, filtering transient symptom descriptions, filtering medical inferences that have not been confirmed by the user, and filtering items that have been saved by the reminder tool. S63. The filtered candidate content is categorized and saved to the current user's long-term memory; the categories include at least: user information, behavioral feedback, stage project, and reference location.

[0013] In some embodiments, step S5, which involves managing the dialogue based on the current user identity, specifically includes: The speech output uses streaming synthesis and playback; during playback, it monitors in real time whether there is speech input that meets the energy threshold and whether the input contains interruption keywords; if a valid interruption keyword is detected, the current speech playback is stopped immediately; the text content or playback ratio that has been played is recorded; before the next round of dialogue begins, the dialogue history is reconstructed based on the actual content that has been played. In step S6, the asynchronous long-term memory extraction is performed by an independent background thread; for the same user, if a new session snapshot is received during the extraction task, the new session snapshot is saved as the latest session snapshot, and the extraction task is restarted immediately after the current task ends based on the latest session snapshot.

[0014] Secondly, the present invention provides an interactive system for multi-user companionship scenarios, comprising: processor; A memory on which computer programs are stored; When the computer program is executed by the processor, the interaction method described in the first aspect is implemented.

[0015] Thirdly, the present invention provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the interaction method described in the first aspect.

[0016] Fourthly, the present invention provides an identity recognition device for an interactive system oriented towards multi-user companionship scenarios, comprising: The identification information acquisition module is used to acquire the identity identification information associated with the current user; The sample library stores identity sample data of registered users; The matching degree calculation module is used to calculate the matching degree between the identity information and each sample in the sample library, and to find the best candidate user and the second candidate user. The determination module is used to determine whether to lock the user's identity based on the predetermined identity determination conditions, and output an identity lock command or a default isolation command based on the determination result; The identity information includes biometric information and / or digital credential information; the predetermined identity determination conditions include: determining whether the matching degree and the matching degree difference between the best candidate user and the second candidate user meet the identity locking conditions based on the matching degree of the best candidate user and the matching degree difference between the best candidate user and the second candidate user.

[0017] Fifthly, the present invention provides a hierarchical memory management device for an interactive system oriented towards multi-user companionship scenarios, comprising: The short-term memory maintenance module is used to maintain a rolling update of session state within a single active cycle, including recent conversation history, compressed summaries, and open items; The long-term memory index module is stored in user-isolated manner and is used to store the subject index for long-term memory. The memory injection module is used to inject the contents of the short-term memory maintenance module and the index of the current user in the long-term memory index module in a layered manner during each round of dialogue construction prompts, and load specific topic details by index only when needed; For unidentified users, the memory injection module will not inject any long-term memory indexes and topic details of registered users.

[0018] Sixthly, the present invention provides an asynchronous memory retrieval device for an interactive system oriented towards multi-user companionship scenarios, comprising: The snapshot receiving module is used to receive session snapshots at the end of the active period; The rule filtering module is used to filter out content in session snapshots that should not be converted into long-term memory based on predefined hard rules; The classification and storage module is used to classify the filtered candidate content and store it in the long-term memory space of the corresponding user; The concurrency control module is used to serialize the memory retrieval tasks according to users. When a new snapshot of a user is received while processing a user's task, the new snapshot is saved and a new round of tasks is started based on the new snapshot after the current task ends.

[0019] The beneficial effects of this invention compared to the prior art include: This invention employs a technical solution of "active cycle identity locking + identity determination based on identity information + default isolated identity." First, during the wake-up phase, a strict identity determination is performed based on predetermined identity determination conditions (e.g., determining whether identity locking conditions are met based on comprehensive matching). Even if the determination is successful, the identity is locked within an "active cycle," fundamentally solving the problem of frequent identity fluctuations and misjudgments in multi-user scenarios. Furthermore, since the identity remains unchanged within the active cycle, the system can perform subsequent dialogue management, memory injection, and tool invocation based on this fixed identity, ensuring the continuity and personalization of interaction and avoiding service interruptions or chaos caused by identity changes. Simultaneously, when the identity determination conditions are not met, the system will not risk binding to the most similar user but will automatically adopt a "default isolated identity." This identity has no right to access the private memories of any registered user, effectively preventing privacy leaks due to identification errors.

[0020] Specifically, the aforementioned technical features are closely synergistic: First, the identity determination conditions enable highly reliable identity recognition during the wake-up phase, providing a prerequisite for identity locking within the active period, thus solving the identity jitter problem and achieving the beneficial effect of robust identity recognition. Second, based on identity locking within the active period, it ensures that all subsequent interactive operations are based on a fixed user identity, thereby achieving strict memory isolation and achieving the beneficial effect of user privacy and security. Finally, while ensuring identity stability and privacy security, the interactive system can efficiently manage dialogue, inject memories, and invoke tools based on a fixed identity, ensuring the continuity and fluency of interaction and achieving the beneficial effect of continuous interaction. These three beneficial effects are mutually supportive and indispensable, collectively constituting the overall technical effect achieved by this invention that balances robust identity recognition, user privacy and security, and continuous interaction.

[0021] In summary, the present invention, through the synergy of the aforementioned technical features, achieves an overall technical effect that balances robustness of identity recognition, security of user privacy, and continuity of interaction in multi-user scenarios on shared devices. This is something that cannot be achieved by relying on a single technical feature (such as using voiceprint recognition alone or using user profiles alone).

[0022] In addition, some embodiments also have the following beneficial effects: Furthermore, by employing voiceprint features for identity determination and introducing specific steps such as "calculating the similarity matrix," "taking the maximum value as the score," and "dynamically adjusting the threshold," the identity recognition process becomes more accurate and reliable. It can adaptively adjust in noisy environments or under different privacy modes, thereby improving the robustness and flexibility of the system.

[0023] Furthermore, by adopting facial features as biometrics and implementing "layered memory injection" (including current round, recent history, compressed summary, long-term index, and on-demand topic details), the model context length is effectively managed. This preserves the continuity of the current task and enables long-term memory access across sessions. At the same time, for unidentified users, the injection of private memory is strictly prohibited, ensuring information isolation.

[0024] Furthermore, by defining the specific steps of "asynchronous long-term memory extraction," especially the filtering-then-classification mechanism based on "predefined filtering rules," the deterministic rules that can be executed by machines are combined with the semantic understanding capabilities of LLM, effectively filtering out junk information that should not be stored, such as transient, sensitive, and unconfirmed information, thereby improving the quality and security of long-term memory.

[0025] Furthermore, by implementing concurrency control strategies of "serialization by user" and "rerun merging" during the asynchronous extraction process, concurrent write conflicts in the same user's memory space are effectively avoided, ensuring the consistency and integrity of the memory data, while also guaranteeing the smoothness of front-end user interaction.

[0026] Furthermore, by providing "classification error handling" and "fallback reporting" mechanisms for tool calls, the system can provide clear and user-friendly feedback when encountering anomalies such as parameter errors, insufficient permissions, or network timeouts, rather than directly crashing or outputting meaningless error messages, thereby improving system stability and user experience.

[0027] Furthermore, by implementing the "streaming speech synthesis and interruption" function and recording the proportion of played content, users can interrupt excessively long or erroneous broadcasts at any time, and in subsequent conversations, they can reconstruct the context based on the actual content heard, greatly improving the naturalness and controllability of voice interaction.

[0028] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0029] Figure 1 This is a flowchart of an interaction method for multi-user companionship scenarios according to an embodiment of the present invention.

[0030] Figure 2 This is a schematic diagram illustrating the overall operation flow and coordination of peripheral asynchronous processes of an interactive system for multi-user companionship scenarios in an embodiment of the present invention.

[0031] Figure 3 This is a state transition diagram of the hibernation / wake-up state machine in an embodiment of the present invention.

[0032] Figure 4 This is a timing diagram of identity determination, memory extraction concurrency, and streaming interruption in an embodiment of the present invention.

[0033] Figure 5 This is a schematic diagram of the active single-turn dialogue, tool orchestration, and voice interruption process in an embodiment of the present invention.

[0034] Figure 6 This is a schematic diagram of the tool anomaly classification and fallback reporting process in an embodiment of the present invention.

[0035] Figure 7 This is a schematic diagram of the process of short-term memory hierarchical injection and long-term memory on-demand retrieval in an embodiment of the present invention.

[0036] Figure 8 This is a schematic diagram of the process for extracting, classifying, deduplicating, and persisting long-term memories for individuals, as described in an embodiment of the present invention. Detailed Implementation

[0037] The present invention will be further described below with reference to the accompanying drawings and preferred embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0038] The core concept of this invention lies in dividing the interactive lifecycle of a multi-user shared device into multiple "active cycles," each starting from user wake-up and ending with user farewell or idle timeout. At the "wake-up boundary" of a cycle, strict identity determination based on identity information (e.g., dual-threshold biometric determination, multimodal fusion determination, or determination based on digital credentials) binds the entire cycle to a defined user identity (or an isolated identity without permissions). Within this cycle, all interactions, memories, and tool calls are based on this fixed identity, thereby fundamentally solving the problems of identity jitter and cross-user privacy leakage. Simultaneously, through mechanisms such as hierarchical memory management, asynchronous task processing, and concurrency control, the smoothness of interaction, the efficiency of context, and the robustness of the system are ensured.

[0039] Based on the above concept, the embodiments of the present invention aim to solve the following problems: In shared devices, when users take turns using the device, how can identity binding be completed during the wake-up phase and maintained during continuous conversations, while avoiding accidental binding when unreliable?

[0040] How to compress the current session goal, open items, and recent semantic rounds into short-term memory, and then inject them hierarchically with long-term memory across sessions, while controlling the context length.

[0041] How to asynchronously extract durable information from completed sessions, deduplicate and update it by individual and semantic type, without blocking foreground voice interaction.

[0042] How to ensure that concurrent activities such as model tool calls, streaming voice output, user interruption, reminder broadcasting, idle sleep, and active recognition do not interfere with each other.

[0043] How can we ensure that unidentified users cannot access the private memories of registered users, and how can we verify outdated information such as medical records, medications, care arrangements, and the location of items before they are used?

[0044] Therefore, the core innovative principle of this invention can be summarized as "identity locking and hierarchical memory injection based on active cycles": First, at the wake-up boundary, an active cycle is bound to a trusted user using voiceprint or facial recognition, and the identity is not repeatedly changed within that cycle; second, in each round of dialogue, short-term states, long-term indexes, and necessary topic details are injected with a fixed user identity; finally, after the session ends, the background extracts and merges long-term memories by user. Compared with schemes that only identify users and call up profiles, this invention extends the identity boundary throughout the entire system architecture of state machines, memories, reminders, tools, and asynchronous threads; compared with simple long-term memory schemes, this invention limits access boundaries, filtering rules, and concurrency consistency under multi-user shared devices.

[0045] Example 1 This embodiment provides an interaction method for multi-user companionship scenarios, executed by an interaction system, such as... Figure 1 As shown, it includes the following steps: S1, Receive voice input stream; S2. Perform voice activity detection and automatic speech recognition on the voice input stream to obtain text information; S3. When the text information contains a wake word, enter the wake-up state and obtain the identity identification information associated with the current user; S4. Based on the identity information, determine the current user's identity according to the predetermined identity determination conditions; if the determination is successful, lock the current active period to the corresponding registered user identity, and refuse to execute any new identity re-determination request within the current active period until the current active period ends; if the determination fails, use the default isolated identity, which has no right to access any registered user's long-term memory and personalized data. S5. Perform dialogue management, memory injection, and tool invocation based on the current user's identity; S6. When a farewell message or idle timeout event is detected, the current active period ends, the current user identity is released, and a session snapshot is submitted for asynchronous long-term memory extraction. The identity information includes biometric information and / or digital credential information; the predetermined identity determination conditions include: determining whether the identity locking conditions are met based on the comprehensive matching degree between the identity information and the registered user sample.

[0046] In this embodiment, once the current active period is locked to a certain user identity (including a registered user identity or a default isolated identity), the system enters an identity stable state. In this stable state, the system refuses to execute any new identity re-determination requests until the current active period ends through step S6. This fundamentally avoids the problem of user identities frequently switching during continuous dialogue due to environmental noise, interruptions from others, or short-term audio interference.

[0047] Meanwhile, for the default isolated identity used due to the failure to meet the identity determination conditions, the system is configured to not have the permission to access any registered user's long-term memory, personalized data, and contextual information. This ensures that data between different users is strictly isolated and protects user privacy even when the user's identity cannot be reliably identified.

[0048] In this embodiment, the identity identification information is preferably a voiceprint feature, the registered user sample is preferably a voiceprint sample embedding vector, and the identity determination condition is preferably a dual threshold condition, specifically including: the similarity between the best candidate user and the second candidate user reaches a first threshold, and the similarity difference between the best candidate user and the second candidate user reaches a second threshold.

[0049] The steps S1 to S6 described above constitute the complete technical solution of this embodiment. To more clearly illustrate the implementation of this technical solution in a specific scenario, the following will elaborate on the above steps in detail from five aspects, including the overall system structure and operation process, multi-user identity locking for active cycles, active single-turn dialogue and tool arrangement, hierarchical generation and injection of short-term memory, and long-term memory extraction and classification for individuals, in conjunction with the interactive system provided in this embodiment.

[0050] I. Overall Structure, Operation Flow, and Asynchronous Process Coordination of the Interactive System This section corresponds to steps S1 and S2, and provides runtime environment support for asynchronous long-term memory extraction in step S6.

[0051] Specifically, after startup, the interactive system of this embodiment first enters a low-power sleep state. In this state, the system continuously runs the voice acquisition and preprocessing module, executing step S1, which involves receiving voice input streams from the environment via a microphone array. Simultaneously, the system runs the streaming speech recognition module, continuously executing step S2, which involves performing voice activity detection (VAD) and automatic speech recognition (ASR) on the received voice input stream to determine if a valid voice command exists. Furthermore, this section describes how the system asynchronously processes submitted session snapshots through a background thread (such as a long-term memory retrieval thread) after the active period described in step S6 ends, thereby decoupling foreground interaction from background memory processing and ensuring smooth system operation.

[0052] The interactive system of this embodiment includes the following functional modules: a voice acquisition module for receiving voice input streams; a voice recognition module for detecting voice activity and automatically recognizing voice input streams; a state machine module for managing state transitions between sleep, wake-up, active, and sleep states; an identity recognition module for determining user identity based on biometrics upon wake-up; a dialogue orchestration module for managing dialogues and calling tools during active periods; a memory module for managing short-term and long-term memory; and a voice output module for streaming speech synthesis and playback.

[0053] like Figure 2 As shown, as a specific implementation of the above modules, the interactive system in this embodiment includes a voice acquisition and preprocessing module, a streaming speech recognition module, a sleep / wake state machine, a multi-user identity module, a dialogue and tool orchestration module, a short-term memory module, a long-term memory module, a reminder and task module, a knowledge and perception tool module, a streaming speech synthesis module, and an expression display module. Specifically, the system consists of a main state machine link on the left and peripheral asynchronous processes on the right, which coordinate their operation through events, state snapshots, and shared queues. The main state machine on the left goes through the following steps in sequence: M1, Startup Initialization: Credentials, Models, User Library, Reminders and Memory Coordinator; M2 and HIBERNATE sleep mode: retain microphone, VAD (Voice Activity Detection), and wake up ASR (Automatic Speech Recognition); release user identity. M3, Voice Triggering and Recognition: Pre-recorded audio, AEC (Acoustic Echo Cancellation), VAD, Streaming ASR, Wake-up Word Detection; M4. Identity Recognition and Locking: Voiceprint / Face Comparison; If it fails, enter the default isolation state, and M7 returns to M2; M5, ACTIVER dialogue loop: security check, memory injection, LLM (Large Language Model) / tools / TTS (Text-to-Speech). M6, Session Closure: Say goodbye or idle timeout; commit snapshot, clear short-term status; M7, Return to Hibernation: Release identity, pause high-resource consumption, and resume listening.

[0054] The corresponding asynchronous processes on the right include: A1, Remind scanning thread; corresponding to M2 / M5: can scan all due reminders while in sleep mode, only scans the current user while in active mode; avoids dialogue / music before broadcasting; A4, Active Face Wake-up Thread; Corresponding to M4: Sleep Delay Low Frame Resumption of Visual Resumption; Enters Identity Locking When a Registered User is Recognized; A5, TTS interrupt listening; corresponding to M5: energy gating during playback; stop playback and record the percentage played when a keyword is hit; A2, Sleep Watchdog; Corresponding to M6: Periodically reads last_user_activity; Submits cleanup event upon idle timeout; A3, Long Memory Extraction Thread; Corresponding to M7: Receive Session Snapshots; Serial by User; Rerun Merge and Atomic Write.

[0055] Among them, M2 and A1 interact through reminder scanning / broadcasting strategies, M4 and A4 interact through proactive event recognition, M5 and A5 interact through interruption events, M6 and A2 interact through idle timeout events, and M7 and A3 interact through session snapshots. Figure 2 In the diagram, blue represents the main state machine, gray represents peripheral asynchronous processes, and arrows indicate event triggering, state reading, or session snapshot submission.

[0056] The interactive system in this embodiment also includes multiple asynchronous processes to support the parallel operation of background tasks and foreground interactions. Specifically, these include an alert scanning thread, a sleep watchdog thread, a long-term memory retrieval thread, an optional active face wake-up thread, and a TTS interruption listening process. These asynchronous processes coordinate with the main state machine through shared state flags and locking mechanisms to ensure data consistency and operational mutual exclusion during multi-task concurrency.

[0057] To ensure multi-task concurrency safety, this embodiment protects the shared state between multiple asynchronous processes using mutex locks or reentrant locks. The shared state includes at least: the current user identifier, system running state, speaker occupancy status, idle timer status, TTS interruption status, and session snapshot queue. Specifically, the protection scope of each type of lock is as follows: the state lock protects the reading and writing of the current user identifier and running state; the audio lock protects the mutual exclusion access to the speaker occupancy flag and the player object; the TTS interruption lock protects interruption-related variables and played text records; and the long-term memory coordination lock protects the session snapshot queue and version number according to the user, ensuring the sequential execution of extracted tasks.

[0058] After system startup, the system sequentially checks model service credentials, loads the reminder module stored by user, creates a long-term memory asynchronous extraction coordinator, loads the local speech activity detection model, and initializes the network retrieval, local medical knowledge base, user database, face recognition, voiceprint recognition, and camera capture objects. Subsequently, a streaming speech recognition session factory is created, registering wake words, domain words, and registered usernames as hot words to the recognition model; background processes such as AEC, sound source direction, reminder scanning, sleep watchdog, and active face wake-up are started according to configuration. If an optional module is unavailable, the reason is recorded and the system is downgraded to a mode where the remaining modules can run.

[0059] The main state machine enters sleep mode by default. While in sleep mode, the system retains only essential listening capabilities, such as microphone acquisition, voice activity detection, and wake word recognition; power-intensive or resource-intensive capabilities such as camera and large-scale model inference can be paused. The system continuously checks for sounds resembling "a person speaking" in the environment: it considers both voice activity detection results and whether the sound energy is significantly higher than the current ambient noise, requiring this state to persist for a short period before recording begins. To prevent the wake word from being truncated, the system also retains a short pre-recorded audio segment before the speech begins; if continuous silence or an excessively long sentence is detected, the current recording session ends.

[0060] ASR sessions are pre-established at the start of speech, with audio blocks being captured and uploaded simultaneously or sent to the local recognizer. At the end of speech, a small amount of silence is appended before asynchronous recognition is completed; if only incremental text is available within a short waiting window, it is used first, and if the final text arrives late, it can be delivered via a queue. Statements without reliable transcription are skipped and do not enter memory, permission checks, or large models. The dormant state only enters identity recognition when the transcribed text contains a wake word; the remaining text after the wake word continues processing as a single command.

[0061] After identity verification, the state machine enters an active dialogue and assigns this active cycle to a specific current user. Subsequent dialogue rounds use this fixed identity for security checks, short-term and long-term memory injection, model inference, tool calls, streaming speech output, and facial expression feedback. When the user says goodbye, or the sleep watchdog determines that idle timeout has occurred, the main thread saves a snapshot of the current session and short-term state, delegates the long-term memory extraction task to the background, clears the model history, session transcription, and short-term state, releases the current user identity, pauses available resources, and returns to sleep mode. If network recognition, the model, speech synthesis, knowledge base, camera, or tools are unavailable, the system degrades by module; long-term memory extraction failures only record alarms and do not affect foreground sessions or existing memory files.

[0062] The peripheral asynchronous processes coordinate with the main state machine through state flags and locks: The reminder scanning thread prioritizes scanning only the current user's reminders during active conversations; when dormant, it can scan all users' due reminders and announce them with their names, or record only the due status according to a privacy policy; the dormant watchdog periodically checks the last user activity time and triggers only when the user is not dormant, the main wheel is not processing, and the user has not spoken; the long-term memory extraction thread processes session snapshots in user order and merges newly generated requests during extraction into the next processing; the active face wake-up thread resumes the camera at a low frequency after a set dormant period, and directly enters identity locking upon recognizing a registered user; during voice broadcasting, interruption monitoring first determines if there is actually a voice, then identifies the interruption keyword, stops broadcasting upon detection, and records the percentage of broadcasts already recorded.

[0063] like Figure 3 As shown, the sleep / wake-up state machine includes seven states: startup initialization, sleep state, wake-up detection, identity recognition, active state, farewell / closing, and release back to sleep. The sleep state enters wake-up detection when the VAD detects continuous speech and the ASR transcription contains a wake-up word; if the voiceprint or face recognition meets the identity determination criteria, it enters identity recognition, loads the user database, and locks the current user; if recognition fails, it enters the active state with a default isolated identity. The active state enters farewell / closing upon receiving a farewell word or when idle timeout occurs, completing session snapshot submission, short-term memory clearing, and identity release before returning to the sleep state. Specifically, the system state machine sequentially goes through: G1. Startup Initialization: Loading models, user library, alerts, and background threads; G2, Sleep Mode: Retain microphone / VAD / awaken ASR; Release identity; G3, during wake-up detection: VAD detects speech; ASR determines whether a wake word is present; G4. Identity Recognition: Loading user database; Voiceprint / Face dual-condition judgment; G5, Active State: Identity locked; Dialogue, Tools, Memory, TTS loop; G6, Farewell / Closing: Received farewell message or idle timeout; Submit session snapshot; G7, Release and Return to Hibernation: Clear short-term status; release identity; pause high-cost resources.

[0064] The process includes related branches and interactions: if G3 determines otherwise, it ignores the content, continues listening, and returns to G2; G4 has an optional branch (registered faces are detected after a sleep delay and directly enter identity locking); G5 interacts with the state of not saying goodbye and not timed out (recording semantic rounds and resetting the idle timer); after G7, it returns to G2. Triggering events include VAD voice, wake-up word, successful voiceprint / face recognition, goodbye word, and idle timeout; actions include starting ASR, locking identity, submitting snapshot, clearing short-term memory, and releasing identity.

[0065] Regarding the locking mechanism, this embodiment uses `threading.Lock` and `threading.RLock` in the Python implementation; in C / C++ or embedded implementations, they can be replaced with `pthread_mutex_t` or an equivalent mutex. State locks protect states such as `robot_state`, `current_user_id`, `current_user_name`, `last_user_activity`, `hibernate_entered_at`, and `session_snapshot`. Locks are acquired before reading or modifying the current user identity, idle timer, and sleep state, and released immediately after the update is complete. Audio locks protect `speaker_occupied_flag` and the player object, preventing the main conversation, notification announcements, and music playback from simultaneously occupying the speaker. TTS interruption locks protect variables such as `interrupt_enabled`, `interrupt_fired`, `played_text`, `played_ratio`, and `scanner_session`. Long-term memory coordination locks protect `pending_snapshot`, `running_flag`, `memory_version`, and `latest_request_time` by user, ensuring that session snapshots, index updates, and topic file writes do not overlap.

[0066] The preferred locking locations are as follows: A state lock is acquired when writing the current user's identifier and status after successful identity verification; a state lock is acquired when the hibernation watchdog reads the idle timer and status; a state lock is acquired when the alert scanning thread reads the current status and decides to scan the current user or all users, and an alert storage lock is acquired when actually reading and writing the alert file; an audio lock is acquired before TTS playback begins and released upon termination or abnormal exit; a memory lock for the user is acquired before extracting the topic file and index from long-term memory, written to a temporary file, verified for version, and atomically replaced before being released.

[0067] II. Multi-user identity locking and permission boundaries based on active periods This section details the specific implementation methods of steps S3 and S4.

[0068] When the text information obtained in step S2 contains a preset wake word, the system enters the wake-up state described in step S3 and immediately obtains the identity information associated with the current user. The identity information includes biometric information (such as voiceprint or facial features) and / or digital credential information (such as temporary tokens transmitted from the user's mobile phone via NFC, Bluetooth, or QR code, device pairing codes, etc.). Subsequently, this section describes the execution process of step S4 in detail: the system compares the extracted biometric information with the biometric sample database of registered users and applies preset dual threshold conditions (i.e., the similarity between the best candidate user and the second candidate user reaches the first threshold T1, and the similarity difference reaches the second threshold T2) to determine the current user's identity. If the determination is successful, the current active period is locked to the corresponding registered user identity; if the determination fails, a default isolated identity is used, thereby achieving high robustness and privacy protection in the identity recognition process.

[0069] In the voiceprint implementation, the system first organizes the wake-up voice into segments suitable for comparison, then extracts voiceprint features that represent the speaker, and compares them one by one with the voiceprint samples of registered users. The system not only determines "who it most resembles," but also whether this candidate is sufficiently credible and significantly superior to other candidates. Only when the highest candidate reaches a preset level of credibility and is sufficiently different from other candidates is the current active period bound to that user. If the conditions are not met, the system maintains the default isolated identity rather than forcibly binding it to the "most similar" person.

[0070] In the identity verification stage, this system adopts a comprehensive matching degree judgment based on identity identification information. Specifically, the system matches the acquired identity identification information with registered user samples, calculates the matching degree, and decides whether to lock the user's identity based on preset identity verification conditions (e.g., dual threshold conditions, probability model thresholds, or machine learning classifier decision boundaries). The following uses voiceprint features as an example to explain in detail a preferred implementation method: dual threshold judgment.

[0071] In this embodiment, the identity locking threshold is implemented through a dual-threshold condition, specifically including a first threshold and a second threshold. The first threshold is used to determine whether the absolute matching degree of the best candidate user is high enough, and the second threshold is used to determine whether the relative difference between the best candidate user and the second candidate user is large enough. Both constitute the judgment criterion for the identity locking threshold. In this embodiment, the voiceprint model is locked as the CAM++ speaker representation model, outputting an L2-normalized speaker embedding vector; each user in the registration database retains 3 to 5 wake-up phrase samples. The system uses 16 kHz mono audio as input, first performing silence trimming and energy gating, and then feeding the wake-up speech into the CAM++ model to obtain the vector e_wake. For the j-th registration sample vector e_ij of user u_i, the cosine similarity s_ij = cos(e_wake, e_ij) is calculated, and all s_ij form a similarity matrix S. The user-level score score_i is the maximum or mean weighted value of the similarity of multiple registration samples for that user.

[0072] Threshold dynamic adjustment: The threshold value is calibrated based on the FAR / FRR curve of the CAM++ model on the sample data of this prototype. In this embodiment, the highest candidate similarity threshold T1=0.65 and the candidate difference threshold T2=0.15 are used. When the ambient noise is consistently higher than the preset noise threshold, or when the device is operating in a care mode with higher privacy requirements, T1 can be increased to 0.75 and T2 can be increased to 0.20. The purpose of this is not simply to pursue the highest recognition rate, but to prioritize reducing the risk of mistaking user A for user B in a shared device.

[0073] The following is a pseudocode implementation of a dual-threshold identity determination based on voiceprint features, as a specific example of the identity determination conditions. In other embodiments, the identity determination conditions can also be implemented using probabilistic models (such as probability thresholds output by a Bayesian classifier), multimodal fusion score weighted sums, or signature verification based on digital credentials.

[0074] The pseudocode for identity determination is as follows, where the input fields include wake-up audio (wake_audio), enroll_db (registration database), first threshold T1, second threshold T2, and default isolation identity (DEFAULT_ID); the output fields include locked_user_id, confidence, margin, and decision.

[0075] def lock_identity(wake_audio, enroll_db, T1=0.65, T2=0.15): e_wake = campp_encoder(wake_audio) # L2-normalized embedding S = {} # cosine similarity matrix user_scores = {} for user_id, samples in enroll_db.items(): S[user_id] = [cosine(e_wake, e_ij) for e_ij in samples] user_scores[user_id] = max(S[user_id]) sorted_users = sorted(user_scores.items(), key=lambda x: x[1],reverse=True) best_id, S_max = sorted_users[0] S_second = sorted_users[1][1]if len(sorted_users)>1 else 0.0 delta = S_max - S_second If S_max >= T1 and delta >= T2: return {"locked_user_id": best_id, "confidence": S_max, "margin": delta, "decision": "LOCKED"} return {"locked_user_id": DEFAULT_ID, "confidence": S_max, "margin": delta, "decision": "DEFAULT_ISOLATED"} The above structure also provides the cosine similarity matrix S, the highest score S_max, the second highest score S_second, and the difference Δ = S_max - S_second. The system only locks the registered user if both S_max ≥ T1 and Δ ≥ T2 are true; otherwise, it enters the default isolation state and does not read any registered user's private long-term memory.

[0076] In other facial recognition implementations, the system selects the main face in the image after the camera recovers and compares it with the registration information in the user database. In digital credential-based implementations, the system obtains a temporary token or device pairing code transmitted by the user device via NFC, Bluetooth, or QR code scanning, and compares it with a pre-stored digital credential sample library for verification. Voiceprints, faces, and digital credentials can be used interchangeably or in combination according to priority or scenario; regardless of the recognition method used, they all ultimately lead to the same personal identifier. Different records generated by multiple registrations of the same natural person can be linked to the same personal identifier, thereby sharing their memory and reminder directories.

[0077] Once an identity is locked within an active cycle, subsequent wake-up calls will not be repeatedly recognized to prevent identity fluctuations caused by short voice messages, TV sounds, or interruptions from others. At the start of each round, the system fixes the user identity used in that round. System prompts, memories, reminders, and tools all operate around this fixed identity, avoiding cross-user access caused by changes in the user database during the cycle. When saying goodbye or going into idle sleep mode, the system releases the current user identity and waits for the next recognition.

[0078] The default user is isolated and not in the public memory pool: the system does not inject any long-term memories of registered users into it; when the default user asks private memory questions such as "Do you remember me?", the system directly refuses to read them before the large model is called and prompts the user to complete identity verification first.

[0079] Figure 4 This is a sequence diagram for identity verification, concurrent memory retrieval, and streaming interruption, involving six parties: user, state machine, identity module, dialogue orchestration, TTS / interruption, and memory thread. The interaction flow is as follows: The user sends a wake-up voice to the state machine; the state machine sends a voiceprint / face comparison to the identity module; the identity module outputs a user ID (user identifier) ​​or default and returns it to the state machine; the state machine fixes the identity snapshot for this round and sends it to the dialogue orchestration; the dialogue orchestration executes incremental TTS to TTS / interruption; at this time, the user triggers an interruption word / interruption and sends it to the TTS / interruption; the TTS / interruption returns a stop playback + played percentage to the dialogue orchestration; the dialogue orchestration truncates the previous assistant message; subsequently, the user sends a farewell or idle timeout to the state machine; the state machine submits a session snapshot to the memory thread; the memory thread extracts the snapshot according to the user lock (version number verification to avoid overwrite conflicts); finally, the memory thread returns an updated index to the dialogue orchestration for subsequent on-demand reading. The front-end link uses fixed identity snapshots; the back-end memory extraction only receives session end snapshots and avoids overwrite conflicts through user locks and version numbers.

[0080] III. Active Single-Turn Dialogue and Multi-Step Tool Arrangement This section corresponds to the part of step S5 concerning "dialogue management" and "tool invocation".

[0081] After successfully locking down the user's identity in step S4, the system enters an active cycle. This section describes how the system manages dialogue and invokes tools based on this fixed current user identity in step S5. Specifically, the system constructs prompts around the locked user identity, invokes a Large Language Model (LLM) for intent understanding and response generation, and parses the tool invocation request output by the LLM. The system uses a tool orchestrator to standardize parameters, verify permissions, and handle errors in tool invocations, realizing a multi-step, interactive tool invocation process and ensuring accurate responses to user requests.

[0082] Once in active state, each text undergoes a pre-emptive security check against high-risk instructions and destructive management requests. Upon passing this check, the system constructs a prompt based on the user's established identity for this round. The prompt includes the current time, bot status, current user reminder summary, short-term memory block, current user long-term memory rules and index, tool call constraints, and speech expression rules.

[0083] The large language model uses streaming output. Tool calls: The system incrementally parses content and tool calls: complete sentences from the speak tool or regular responses can be immediately fed into the streaming TTS while the model is still generating subsequent content; after the tool call parameters are incrementally completed, the tool orchestrator parses multiple tools within the same step and uses a thread pool to execute independent tools in parallel. A set of non-repeatable tools is maintained within the same user round, and repeated calls return structured skipped results.

[0084] Each round allows for a multi-step model-tool loop. The model first outputs a brief progress message and invokes the tool; the tool's result is used as a tool message to populate the history; the next step of the model generates the final response based on this. If the model only says "Let me check" without actually invoking the tool, the system adds a correction prompt requiring it to invoke the tool or explicitly end the process; if the tool malfunctions and the model does not verbally explain, the system uses a user-understandable message from the structured error to provide a fallback message.

[0085] Tool exception handling, such as Figure 6As shown. Before invoking the tool, parameter normalization and permission verification are performed. Missing necessary parameters, unidentified current user, access to others' memories, or destructive management requests are considered non-retryable errors, directly returning a fixed error code and a user-understandable message. Network timeouts, temporary unavailability of the knowledge base, and temporary failure of the visual service are considered retryable errors, returning a "retryable" flag and allowing the model to retry once in the next step with corrected parameters or a narrower problem. When the same user repeatedly invokes a non-repeatable tool within the same round, a structured result of "executed / skipped" is returned. The fallback message comes from a predefined error code mapping table; for example, a network error corresponds to "I cannot connect to the query service now, you can try again later," and a permission error corresponds to "This content requires verification of the current user's identity," rather than the model fabricating tool results out of thin air. Specifically, the tool exception classification and fallback reporting process proceeds as follows: Tool call request: Carries the fixed user identity and parameters for this round; Parameter and permission verification: Block requests that lack proper means, exceed authority, or are destructive. Execution tools: Set timeout; categorize and capture exceptions.

[0086] After execution, it splits into three branches: Retryable errors: Network timeout, temporary service failure, return the retryable flag; Non-retry error: Insufficient permissions, invalid parameters, user not recognized, returns a fixed error code; Business skip results: For repeated calls or completed tasks, a structured skip reason is returned.

[0087] Then, it is uniformly converted into a user-understandable message: generated by the error code mapping table, preventing the model from fabricating results.

[0088] Finally, if the model fails to report an error, the main loop will directly broadcast a fallback message and terminate or request a retry. The fallback message comes from predefined error codes and user dialogue templates; the LLM (Large Language Model) is only responsible for subsequent organization and not for generating tool results out of thin air.

[0089] Tools include reminders for adding new items, lists, completed and recently completed queries, explicit long-term memory storage and topic memory retrieval, local medical knowledge base retrieval, web search, visual recognition, music playback, user registration and management, etc. For web search, visual, and medical knowledge retrieval, tool types can be mapped to active emoticons, synchronizing progress voice prompts such as "Let me take a look" and "Let me check" with emoticons such as looking, searching, consulting, and recalling.

[0090] When a streaming TTS session is interrupted, the system does not attempt to modify the unfinished hidden states within the larger model. Instead, it reconstructs the explicit dialogue history before the next round of inference. Specifically, the TTS player maintains the text to be played, the text already played, and the percentage of text already played. When the interruption keyword is hit, the stop event cancels subsequent synthesis and playback, truncates the most recent assistant message to the segment the user actually heard, and adds machine-readable markers, such as "The user only heard the first half of the previous reply and did not hear the rest." The next round of the model continues the dialogue only based on this truncated history, thus avoiding repeating unplayed content or continuing inference based on content the user never heard.

[0091] Voice Interruption: In this embodiment, voice output employs streaming synthesis and playback, with a dedicated interruption monitoring thread monitoring audio input in real time. When voice that meets the energy threshold and contains interruption keywords is detected, the system immediately stops the current voice playback. After an interruption occurs, the system records the text segment actually output through the TTS player. Before the next round of dialogue begins, the system reconstructs the dialogue history based on the actually played content, allowing the model to continue interacting based on what the user actually heard, rather than on the unplayed complete response.

[0092] Figure 5 The main dialogue flow demonstrates how tool calls, streaming announcements, and interrupt listening work together within the same active dialogue round. The main dialogue steps (blue) proceed sequentially as follows: F1, End of Voice: Receive text and identity snapshot; F2. Pre-security and permission check: Construct short-term + long-term memory prompts; F3, Large Model Streaming Inference: Maximum Multi-Step Tool Loop; F4, Parsing tool call: parallel execution / duplicate removal / error fallback; F5, Streaming TTS and Emoticons: Sentence-Boundary Incremental Broadcasting; F6, Record Semantic Rounds: Updates short-term session status, and retains the dialogue history in a sliding window.

[0093] F4 triggers the tool loop, the tool returns a result, appends a "tool" message to the next inference step, and flows back to F3. Parallel listening or history correction (gray) includes: during F5, parallel microphone listening, after energy gating, entering the interruption word ASR scanner, identifying the keyword and setting a stop event, canceling synthesis and recording the proportion of played content, and rewriting the assistant message before the next round, retaining only the actually heard part to prevent the duplication of unplayed content. Figure 5 In the diagram, blue represents the main dialogue steps, gray represents parallel listening or historical correction, and arrows indicate the execution order or loop backfilling.

[0094] IV. Methods for the Stratified Generation and Injection of Short-Term Memory Short-term memory doesn't simply pile up the original messages; instead, it maintains a rolling "conversation state" within the current active period and the current user's scope. After each round visible to the user is completed, the system records the time of that round, the user's request, the assistant's final response, and the necessary tools used, while simultaneously updating the current topic, ongoing goals, unfinished tasks, resolved tasks, timelines, and care risk alerts.

[0095] In the preferred embodiment, the most recent 8 semantic rounds are retained; when the upper limit is exceeded, the earliest 4 rounds are converted into summary entries each time, retaining a maximum of 24 compressed summaries, while only the most recent 12 can be taken during injection. The most recent 4 items are preferred for both open and resolved items, and the most recent 6 items are preferred for both time anchors and care risk labels. All of the above values ​​can be adjusted according to the model context window.

[0096] like Figure 7 As shown, the injection order is from most recent to furthest, and from lightest to heaviest: first, the current fixed identity, the current question, and the most recent original dialogue are added; then, the most recent semantic round, compressed summaries of earlier rounds, and the session state are added; next, the current user's long-term memory index is added; finally, the specified topic file is read only when the user explicitly requests recall, or when the index indicates that a topic is relevant to the current question and requires detail. The long-term memory index only stores "topic file pointer + single-line semantic hook," and does not automatically add all details to the prompt.

[0097] Therefore, each round of prompts does not cram the entire history in at once, but rather pieces it together layer by layer according to "basic rules, current issue, recent conversations, short-term status, long-term index, and topic details when necessary." Content appearing earlier in the list is closer to the current round and updates more frequently; content appearing later is more persistent and has a narrower scope. For unidentified users, the system will not add any long-term indexes or topic details for registered users.

[0098] Specifically, the short-term memory stratified injection and long-term memory on-demand retrieval process includes the following steps: L0, Round Identity and Real-Time Input: Fixed User Identity, Current User Question, Most Recent Original Dialogue Window; L1, Recent Semantic Layer: The most recent semantic rounds, including user requests, assistant conclusions, and optional tool trajectories; L2, Rolling Compression Layer: Earlier rounds that are outside the recent window are compressed into summary entries in batches; L3, Session State Layer: Topic, Current Goal, Open Items, Resolved Items, Time Anchor, Care Risk Label; L4, Personal Long-Term Index Layer: Only injects the long-term memory index of the current user; L5, Topic Details Layer: The tool is invoked to read the specified topic file only when the issue is relevant or the user clearly recalls it.

[0099] The injection principles are as follows: the higher up the injection point, the closer it is to the current cycle and the higher the update frequency; the lower down the injection point, the more persistent it is and the narrower the reading range.

[0100] Medical information, medications, allergies, appointments, care arrangements, and item locations may change; long-term memory is only considered as historical statements that the user previously stated or confirmed when the information was saved. Before providing suggestions or taking actions based on such memories, the system rechecks with a confirmation question or "You said this before." When the current session conflicts with old memories, the current input takes precedence, and the old memory is updated during the session-end extraction phase. The triggering of confirmation questions is selective; not all memories require confirmation. The system prioritizes confirming high-risk memories (such as medical, medication, and financial information), while using low-risk information (such as the user's preference for "liking to drink black tea") directly.

[0101] V. Personalized Long-Term Memory Retrieval and Classification Methods This section corresponds to step S6 regarding "submitting a session snapshot for asynchronous long-term memory extraction," and provides a detailed explanation of the core feature of "asynchronous long-term memory extraction" in this step.

[0102] When step S6 detects a farewell message or an idle timeout event, the system terminates the current active cycle and submits a session snapshot. This section details the asynchronous processing of this session snapshot in the background. The system first receives the submitted session snapshot, then filters out candidate content that should not be saved as long-term memory based on predefined hard filtering rules (such as filtering transient symptoms, sensitive identifiers, and unconfirmed medical inferences). Finally, the filtered candidate content is categorized and saved to the current user's long-term memory, including categories such as user information, behavioral feedback, stage items, and reference locations. This section also describes how mechanisms such as "rerun merging" and version number verification ensure the concurrency safety of the asynchronous extraction task.

[0103] The long-term memory module comprises a storage submodule, an extraction and classification submodule, an index maintenance submodule, and an asynchronous coordination submodule. Long-term memory is stored separately for each user, with each user having an independent memory space, including a long-term memory index and several topic files. The index tells the model "what is worth remembering," and the topic files store specific facts, sources, reasons, and applicable methods. During writing, a temporary file is written first, and then the target file is updated using atomic replacement, reducing half-writes caused by power outages or concurrency.

[0104] This embodiment of the invention sets up two entry points: explicit writing and session end extraction. Explicit writing is used for persistent facts that the user directly requests "remember," and the system will write the fact into the current user's long-term memory. Session end extraction is used for user farewell or idle hibernation: the main thread first takes a snapshot of the session and short-term state, then submits the task to an independent event loop thread, and the foreground immediately completes hibernation.

[0105] The extractor only processes the user's most recently completed sessions, while also reading the names and summaries of existing topics in the user's memory, short-term session summaries, and the current absolute time, allowing the model to generate a candidate list. The decision order is: first, determine if it is worth storing long-term; then, determine the category; and finally, determine whether to create a new session or update an existing one. Instantaneous chatter, one-off symptoms, camera inferences, model-inferred diagnoses, passwords or ID numbers, original biometrics, other user information, and items already saved by the reminder tool are not included in long-term memory.

[0106] To reduce the uncertainty of LLM extraction, this embodiment sets machine-executable hard filtering rules before and after model judgment. Candidate memory objects include at least the following fields: text, subject_user_id, source_user_id, entity_type, time_span_hours, is_user_confirmed, contains_biometric_raw, contains_identifier, from_reminder_tool, and memory_type. The system first records the user identifier locked during the current active period as current_user_id, and then checks each candidate item to determine whether it is allowed to enter long-term memory.

[0107] SYMPTOM_RE = r"(pain|ache|dizziness|cough|cough|fever|nausea|chest tightness|shortness of breath|diarrhea|insomnia)" UNCERTAIN_RE = r"(Possible|Suspected|Probably|Speculate|Consider|Like|Maybe|Not Ruled Out)" IDENTIFIER_RE = r"(\d{17}[\dXx]|\d{11}|password|verification code|bank card|ID card)" def allow_long_memory(c, current_user_id, authorized_care_ids): if c.subject_user_id not in {current_user_id, *authorized_care_ids}: return False, "OTHER_PERSON" if re.search(IDENTIFIER_RE, c.text) or c.contains_biometric_raw: return False, "SENSITIVE_IDENTIFIER" if c.entity_type == "symptom" and re.search(SYMPTOM_RE, c.text) andc.time_span_hours<= 24: return False, "TRANSIENT_SYMPTOM" is_uncertain_medical = ( c.entity_type in {"diagnosis", "medical_conclusion"} and re.search(UNCERTAIN_RE, c.text) and not c.is_user_confirmed ) if is_uncertain_medical: return False, "UNCONFIRMED_INFERENCE" if c.from_reminder_tool and c.memory_type == "reminder_text": return False, "DUPLICATED_REMINDER" return True, "ALLOW" For example, if the current user is user_A, and the candidate "Dad is a little dizzy today" resolves to user_B and user_B is not an authorized care object of user_A, then OTHER_PERSON is returned directly; if the candidate "I had a cough this afternoon" has a time_span_hours less than or equal to 24 hours, it is only retained in the current session state and not written to long-term memory; the candidate "I confirm that I am allergic to penicillin" can be entered into the user class or reference class for long-term memory because the subject is the current user and it belongs to a fact that the user has clearly confirmed.

[0108] The four memory categories are as follows: The user category stores the user's identity, title, stable preferences, abilities, lifestyle habits, and care background explicitly stated or confirmed by the user; the feedback category stores the user's corrections or effective confirmations of the robot's behavior, and records the why and how to apply; the project category stores time-bound plans, follow-up visits, rehabilitation, family or doctor coordination, and support arrangements in case of changes, and converts relative dates to absolute dates; the reference category stores stable locations and pointers for medications, documents, instructions, hospitals, or service channels.

[0109] If a compatible category memory already exists for the same topic and the same user, the original file and index entries are updated, and no duplicate creation is performed. If a new statement conflicts with an old memory, the current user's statement takes precedence, and the old file is updated. The coordinator executes sequentially for the same user: when an existing task is running, the latest session snapshot is retained, and the task is processed again after the current task ends; different users can queue up separately, thus avoiding concurrent write conflicts in the same memory space without blocking the main speech thread.

[0110] To ensure consistency during reruns and merges, this embodiment maintains update times or version numbers for each user's memory index and topic files. At the start of the extraction task, the current version is read. After generating candidate updates, the version is read again before being written to the database: if the version hasn't changed, it's written to a temporary file and atomically replaced; if the version has changed, it indicates that the previous task or explicit save has written the new facts, and the current task abandons direct overwriting, rereading the latest index, topic summary, and merged session snapshot before making a final judgment. For project-type memories of the same topic, fact timestamps and source priorities are also compared: facts explicitly confirmed by the current user take precedence over older facts, explicit saves take precedence over automatic extraction, and newer absolute times take precedence over older times.

[0111] Personalized long-term memory retrieval, classification, deduplication, and persistence processes, such as Figure 8 As shown, the process proceeds in sequence: Session end snapshot: Includes the current user's semantic round and a short session summary; Read the current user's existing memory names and summaries; Extraction hint: Only extract new or need-to-update durability facts; The large model then provides a list of candidate memories and determines whether to not save, create, or update them.

[0112] Candidate memories are categorized into four dimensions: User: Identity, title, stable preferences, and confirmed care background; Feedback: Correction or confirmation of robot behavior, including reasons and applicable methods; Project: Phase plan, follow-up visits, rehabilitation, and conversion of relative time to absolute time; Reference: A stable location or pointer for items, documents, or service channels.

[0113] After categorization, the data is written into separate topic files: structured description plus main text, which are replaced using atomic temporary files. And update the long-term memory index: add a single-line semantic hook to the file pointer, and update entries with the same name instead of appending them repeatedly.

[0114] Experimental Example The prototype configuration for this experimental example is as follows: the main control board uses an RK3588S edge computing board with 8GB of RAM and 64GB of storage; the microphone uses a ReSpeaker 4-Mic Array with an input sampling rate of 16 kHz; the camera uses a Logitech C920e 1080p RGB camera; and the speaker is a 3W mono speaker with a playback sampling rate of 22.05 kHz. The software environment uses a Linux system, Python 3.10 main control program, CAM++ voiceprint model, RetinaFace-like face detector, ArcFace-like face feature model, streaming ASR interface, LLM tool orchestrator, and streaming TTS interface. If deployed to an Intel i5 or other edge platform, this would be considered a replacement of the computing platform; identity locking, memory isolation, and asynchronous collaboration processes would remain unchanged.

[0115] Identity Recognition Experiment Example: Ten registered users were selected, and 20 wake-up voice recordings were collected from each user. An additional 20 voice recordings of non-users or background television voice recordings were collected as negative samples. Using the CAM++ voiceprint model and setting the highest candidate similarity threshold T1=0.65 and the difference between the highest and second-highest candidate T2=0.15, the prototype achieved a wake-up identity locking accuracy of approximately 96% in a quiet indoor environment, with a false acceptance rate of approximately 1% to 2% and a false rejection rate of approximately 3% to 5%. In a noisy environment, increasing the threshold to 0.75 / 0.20 reduced the false acceptance rate but increased the proportion of default identity isolation. The above data illustrates the moderating relationship between the threshold and privacy risk and usability. The final product should retain the test audio source, environmental noise, sample size, and FAR / FRR statistics table.

[0116] The following is a simplified code snippet of identity writing, TTS mutual exclusion, and version checking memory from this experimental example, demonstrating the lock protection location and atomic update method: state_lock = threading.RLock() audio_lock = threading.Lock() memory_locks = defaultdict(threading.Lock) def enter_active_after_identity(result): with state_lock: robot_state.current_user_id = result["locked_user_id"] robot_state.identity_confidence = result["confidence"] robot_state.identity_margin = result["margin"] robot_state.mode = "ACTIVE" robot_state.last_user_activity = time.time() def speak_stream(chunks): with audio_lock: for text in chunks: if interrupt_event.is_set(): break tts_player.play(text) def update_memory_atomic(user_id, candidate, expected_version): with memory_locks[user_id]: latest = read_memory_index(user_id) if latest.version != expected_version: merged = merge_again(read_pending_snapshot(user_id), latest,candidate) else: merged = apply_candidate(latest, candidate) write_temp_then_replace(user_id, merged) Response latency experiment example: Without streaming ASR, streaming LLM, and streaming TTS, the average time from the end of the user's speech to the first time hearing the robot's feedback is approximately 3.5 to 5 seconds. After adopting the streaming recognition, sentence boundary incremental broadcasting, and tool progress voice of this embodiment of the invention, the first perceived feedback time can be reduced to 1 to 2 seconds. For rounds requiring network retrieval or visual recognition, the system first broadcasts "Let me check / Let me take a look" and links it with facial expressions, then waits for the tool's results. Therefore, the user's perceived waiting time is shorter than that of a purely serial solution.

[0117] Memory extraction experiment example: Using 100 multi-user family companionship sessions as the test set, we compared two methods: "direct injection of full history" and "short-term hierarchical + long-term index on-demand reading". The latter only injects the current user index and necessary topic details in each round of prompts, which can reduce the average prompt length by about 40% to 60%; background extraction after the session ends does not block the foreground sleep, a single extraction failure will not destroy the existing index, and rerun merging and version verification can avoid the overwrite conflict caused by consecutive sessions of the same user.

[0118] Hard filtering rule validation example: After mixing samples containing transient symptoms, unconfirmed diagnoses, other people's information, ID numbers, and casual conversation into the conversation, the rule layer first removes content that is prohibited from being saved, and then feeds the remaining candidates to the model for classification. For candidates in the categories of medical, drug, and care arrangements, if the source is not explicitly stated or confirmed by the user, they are only entered into the current conversation state and not into long-term memory. This design allows the long-term memory module to leverage the semantic understanding capabilities of LLM while maintaining privacy and security boundaries through deterministic rules.

[0119] The embodiments of the present invention have the following advantages compared with the prior art: Complete closed loop: Unify startup, low-interference hibernation, wake-up recognition, continuous dialogue, tool invocation, interruptible broadcasting, automatic hibernation, and memory storage into an executable state machine.

[0120] Identity stability and privacy isolation: Identity is only bound and released at the boundary of the active cycle, and an immutable identity snapshot is used in each round; when unreliable, it enters the default isolation state to reduce cross-user memory and reminders.

[0121] Context efficiency: Recent semantics, earlier summaries, session states, long-term indexes, and topic details are injected layer by layer, which preserves the continuity of the current task and avoids the entire history from filling the model window.

[0122] Memory quality: First determine whether to save, then classify, deduplicate, and update; explicit saving and session-end extraction complement each other, atomic writing and user-sequential coordination reduce duplication and concurrent corruption.

[0123] Companionship safety: Caregiver memories are stored from their source and confirmed before use; current input overwrites old memories; unidentified users cannot access private long-term memories.

[0124] Low latency and perceptibility: ASR, LLM and TTS are processed in a streaming manner, and the progress of voice and facial expressions in the same step reduces the uncertainty of waiting. Keyword interruption avoids long replies from getting out of control.

[0125] Concurrency robustness: The sleep watchdog, active identification and memory extraction are used to run in the background thread or asynchronous process, and the lifecycle of the speaker, identity and session is coordinated with the lock through status flags.

[0126] Alternative solutions that do not deviate from the core concept: Without departing from the core concept of "active cycle identity boundary + hierarchical personal memory + asynchronous collaboration" in the embodiments of the present invention, the following steps may adopt equivalent alternatives, which do not change the substantive protection scope of the embodiments of the present invention.

[0127] 1. Wake-up and hibernation triggering methods: Wake-up can be triggered by a local wake word model, lightweight ASR, button, gesture, near field detection or registered face; hibernation end conditions can be a goodbye word, idle timeout, remote command or power policy.

[0128] 2. Identity Recognition Information and Judgment Rules: Identity identification information can be voiceprint features, facial features, wearable device near-field credentials, digital tokens transmitted via mobile phone NFC / QR code, PIN codes, etc.; identity determination can be based on matching rules such as cosine similarity, Euclidean distance, probability calibration, multimodal fusion score weighting, or digital signature verification; judgment conditions can be based on double thresholds, single threshold + context verification, probabilistic model thresholds, or machine learning classifier decision boundaries, etc.

[0129] 3. Long-term memory storage method: Long-term memory can be saved as Markdown, JSON, SQLite, relational database, vector database or encrypted cloud database; as long as reading and writing always revolve around the same user identity and the index and details are kept hierarchical.

[0130] 4. Memory classification, extraction, and deduplication methods: Four types of memory can be added, removed, or used to form a hierarchical ontology; extraction can be performed using a large language model, rule classifier, small model, or manual verification; deduplication can be performed using the same-name topic, vector similarity, entity key, or time version.

[0131] 5. Consistency guarantee methods for rerun merge: Consistency of rerun merge can be achieved by optimistic locking version number, pessimistic locking, database transactions, event log replay, or append-only fact stream; as long as it is guaranteed that subsequent extraction is based on the latest set of completed and pending facts.

[0132] 6. Short-term memory generation and window adjustment methods: Short-term memory can be generated using rule features, model summaries, or a combination of both; window size, compression batches, and injection budget can be dynamically adjusted based on model context, response latency, and risk level.

[0133] 7. Tool execution and feedback methods: Tools can be executed sequentially or in parallel; progress feedback can be provided through voice, facial expressions, lights, screen animations, text status, or mechanical actions; interruptions can be made through keywords, buttons, gestures, or acoustic events.

[0134] 8. Model deployment methods: Large language models, ASR, TTS and visual models can be deployed locally, at the edge, in the cloud or in a hybrid manner; degradation and permission boundaries remain unchanged when a module fails.

[0135] The core protection scheme of this invention should be built around the following complementary technical features, which together constitute a complete closed loop from "hibernation and wake-up" to "identity locking" and then to "memory management", none of which can be omitted.

[0136] State machine and identity binding: hibernation-wake-active-hibernation state machine, where the wake-up voice also serves as identity recognition material, and the same command is directly processed after identity is locked.

[0137] Identity verification and privacy isolation: Multi-user identity locking is performed based on identity information (including biometrics and / or digital credentials) and preset identity verification conditions (e.g., dual threshold conditions, probability model thresholds, etc.). If the verification fails, the system enters a default isolation state where no private memories are injected.

[0138] Concurrency control and locking mechanisms: Mutex / reentrant lock protection mechanisms used to coordinate the main state machine, alert scanning, sleep watchdog, TTS interruption, and long-term memory retrieval, as well as the corresponding set of shared state variables.

[0139] Identity stability within an active cycle: an access control method that keeps the identity unchanged within an active cycle, generates an immutable identity snapshot in each round, and releases the identity when dormant.

[0140] Unified identity identification and cross-modal mapping: Using a unified personal identifier to map voiceprints, faces, long-term memory and tool permissions, enabling multiple registration records of the same natural person to be shared while different natural persons are isolated.

[0141] Layered memory injection: A layered memory injection method that includes a recent semantic layer, a scrolling compression layer, a session state layer, a long-term index layer, and an on-demand topic details layer.

[0142] Long-term memory classification and hard filtering: classification of user, feedback, project, and reference memories for companion care, hard filtering rules, filtering before classification, conflict update, and source labeling methods.

[0143] Asynchronous extraction and rerun merging: Asynchronous extraction during farewell or idle sleep, serialized by user and rerun merging for new requests, combined with version number verification, atomic persistence of topic files and indexes.

[0144] Multi-step tool loop and exception handling: a multi-step tool loop that integrates model generation, tool execution, progress voice and facial expression linkage, as well as a classification and fallback reporting mechanism for tool failures, missed calls, duplicate calls, timeouts and permission errors.

[0145] Voice interruption and context consistency: A context consistency method that uses energy gating to perform keyword recognition interruption during TTS and rewrites the dialogue history based on the actual proportion of speech played.

[0146] Furthermore, although this solution significantly reduces the risk of misjudgment through dual threshold determination, it may still occur in extreme situations (such as drastic changes in the acoustic environment or malicious attacks). As a supplementary security mechanism, the interactive system can maintain an internal "identity confidence" score. When, in N consecutive rounds of dialogue, the user's behavior or language pattern is significantly inconsistent with the locked identity (such as address or preferences), the system can proactively request the user to perform secondary identity verification (such as "Please state your name" or "Let me confirm your identity"), thereby maintaining identity stability while providing a security fallback.

[0147] References [1] EP1941411B1, Companion robot for personal interaction, https: / / patents.google.com / patent / EP1941411B1 / en [2] US10452816B2, Method and system for patient engagement, https: / / patents.google.com / patent / US10452816B2 / en [3] US20150314454A1, Apparatus and methods for providing a persistentcompanion device, https: / / patents.google.com / patent / US20150314454A1 / en [4] US11670156B2, Interactive reminder companion, https: / / patents.google.com / patent / US11670156B2 / en [5] Zhong et al., MemoryBank: Enhancing Large Language Models withLong-Term Memory, AAAI 2024, https: / / ojs.aaai.org / index.php / AAAI / article / view / 29946 [6] Packer et al., MemGPT: Towards LLMs as Operating Systems, arXiv:2310.08560, https: / / arxiv.org / abs / 2310.08560 [7] Wang et al., Augmenting Language Models with Long-Term Memory,arXiv:2306.07174, https: / / arxiv.org / abs / 2306.07174 The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, several equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or purpose, should be considered within the scope of protection of the present invention.

Claims

1. An interaction method for multi-user companionship scenarios, characterized in that, Performed by the interactive system, the process includes the following steps: S1, Receive voice input stream; S2. Perform voice activity detection and automatic speech recognition on the voice input stream to obtain text information; S3. When the text information contains a wake word, enter the wake-up state and obtain the identity identification information associated with the current user; S4. Based on the identity information, determine the current user's identity according to predetermined identity determination conditions; if the determination is successful, lock the current active period to the corresponding registered user identity, and refuse to execute any new identity re-determination request within the current active period until the current active period ends; if the determination fails, use the default isolated identity, which has no right to access any registered user's long-term memory and personalized data; the predetermined identity determination conditions specifically include: S41. Calculate the cosine similarity between the voiceprint embedding vector of the wake-up voice and the voiceprint embedding vectors of each registered user's voiceprint sample, and form a similarity matrix; S42. For each registered user, take the maximum similarity among all their voiceprint samples as the score of that registered user. S43. Identify the best candidate user with the highest score and the second-highest candidate user with the second-highest score; S44. Determine whether the score of the best candidate user is greater than or equal to the first threshold, and whether the difference between the score of the best candidate user and the score of the second candidate user is greater than or equal to the second threshold. S45. If so, then the identity of the best candidate user is locked. S46. If not, then the default isolation identity will be used; The values ​​of the first threshold and the second threshold can be dynamically adjusted according to the ambient noise level or the privacy mode set by the interactive system. S5. Perform dialogue management, memory injection, and tool invocation based on the current user's identity; S6. When a farewell message or idle timeout event is detected, the current active period ends, the current user identity is released, and a session snapshot is submitted for asynchronous long-term memory extraction. The identity information includes biometric information and / or digital credential information; the predetermined identity determination conditions include: determining whether the identity locking conditions are met based on the comprehensive matching degree between the identity information and the registered user sample.

2. The interaction method according to claim 1, characterized in that, The identity information is based on voiceprint features.

3. The interaction method according to claim 1, characterized in that, The identity information is facial features, and the registered user samples are a facial image sample library; In step S5, the injected memory is layered, specifically including: The current round of user requests and assistant responses; The original dialogue history of several recent rounds; Compressed summaries of earlier rounds of dialogue; The semantic state of the current session, including open items, resolved items, and timeline; The current user's long-term memory index; Only inject specific topic details specified in the long-memory index as needed, when they are relevant to the current request. Specifically, for current users using the default isolated identity, no long-term memory indexes or topic details of registered users will be injected.

4. The interaction method according to claim 1, characterized in that, The tool invocation described in step S5 includes the following steps: S51. Parse the tool call request output by the large language model; S52. Perform parameter specification and permission verification on the request; S53. Classify and handle errors according to their types: For non-retryable errors such as missing parameters or insufficient permissions, directly return the error code and a user-understandable message. S54. For retryable errors such as network timeout or service unavailability, return a retryable flag; S55. If the large language model does not provide an explanation for the tool call failure, the interactive system will directly broadcast a predefined, user-understandable message corresponding to the error code. The asynchronous long-term memory extraction described in step S6 includes the following steps: S61. Receive the submitted session snapshot; S62. Based on predefined filtering rules, filter out candidate content that should not be saved as long-term memory; the filtering rules include at least: filtering content that the subject user is not the current user and is not their authorized care object, filtering content containing sensitive identifiers or original biometrics, filtering transient symptom descriptions, filtering medical inferences that have not been confirmed by the user, and filtering items that have been saved by the reminder tool. S63. The filtered candidate content is categorized and saved to the current user's long-term memory; the categories include at least: user information, behavioral feedback, stage project, and reference location.

5. The interaction method according to claim 4, characterized in that, Step S5 involves managing the conversation based on the current user's identity, specifically including: The speech output uses streaming synthesis and playback; during playback, it monitors in real time whether there is speech input that meets the energy threshold and whether the input contains interruption keywords; if a valid interruption keyword is detected, the current speech playback is stopped immediately; the text content or playback ratio that has been played is recorded; before the next round of dialogue begins, the dialogue history is reconstructed based on the actual content that has been played. In step S6, the asynchronous long-term memory extraction is performed by an independent background thread; for the same user, if a new session snapshot is received during the extraction task, the new session snapshot is saved as the latest session snapshot, and the extraction task is restarted immediately after the current task ends based on the latest session snapshot.

6. An interactive system for multi-user companionship scenarios, characterized in that, include: processor; A memory on which computer programs are stored; When the computer program is executed by the processor, it implements the interactive method as described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instruction is executed by the processor, it implements the interactive method as described in any one of claims 1 to 5.

8. An identity recognition device for an interactive system oriented towards multi-user companionship scenarios, characterized in that, include: The identification information acquisition module is used to acquire the identity identification information associated with the current user; The sample library stores identity sample data of registered users; The matching degree calculation module is used to calculate the matching degree between the identity information and each sample in the sample library, and to find the best candidate user and the second candidate user. The determination module is used to determine whether to lock the user's identity based on the predetermined identity determination conditions, and output an identity lock command or a default isolation command based on the determination result; The identity information includes biometric information and / or digital credential information; the predetermined identity determination conditions include: determining whether the identity locking conditions are met based on the matching degree of the best candidate user and the difference in matching degree between the best candidate user and the second candidate user.

9. A hierarchical memory management device for an interactive system oriented towards multi-user companionship scenarios, characterized in that, include: The short-term memory maintenance module is used to maintain a rolling update of session state within a single active cycle, including recent conversation history, compressed summaries, and open items; The long-term memory index module is stored in user-isolated manner and is used to store the subject index for long-term memory. The memory injection module is used to inject the contents of the short-term memory maintenance module and the index of the current user in the long-term memory index module in a layered manner during each round of dialogue construction prompts, and load specific topic details by index only when needed; For unidentified users, the memory injection module will not inject any long-term memory indexes and topic details of registered users.

10. An asynchronous memory retrieval device for an interactive system oriented towards multi-user companionship scenarios, characterized in that, include: The snapshot receiving module is used to receive session snapshots at the end of the active period; The rule filtering module is used to filter out content in session snapshots that should not be converted into long-term memory based on predefined hard rules; The classification and storage module is used to classify the filtered candidate content and store it in the long-term memory space of the corresponding user; The concurrency control module is used to serialize the memory retrieval tasks according to users. When a new snapshot of a user is received while processing a user's task, the new snapshot is saved and a new round of tasks is started based on the new snapshot after the current task ends.

Images