An alzheimer's disease patient intelligent assistance method and system based on multi-modal perception and double-layer AI architecture

Abstract

The application discloses an Alzheimer's disease patient intelligent assistance method and system based on multi-modal perception and a double-layer AI architecture. The method comprises the following steps: a wearable terminal collects multi-modal input data and sends the multi-modal input data to an edge computing server; the edge computing server serves as a first computing node, generates a restricted structured service call instruction through a lightweight large language model, and executes a first service; a user question and an execution result are sent to a cloud-side workflow for checking; if the first execution is insufficient to answer the question, the cloud-side workflow outputs a hybrid protocol response containing a natural language carrier and a machine-readable correction load; the edge node extracts the correction load and reuses a frozen multi-modal data segment to perform compensation; and the compensation result is submitted to the cloud side to generate a final reply and broadcast the final reply. Through the double-layer architecture of edge computing and cloud-side workflow cooperation, and in combination with an automatic error correction closed loop mechanism, the application realizes low-delay and high-reliable Alzheimer's disease patient intelligent assistance.

Description

Technical Field

[0001] This invention relates to the fields of smart wearable devices, artificial intelligence, the Internet of Things, and medical assistance technologies, specifically to an intelligent assistance system and method for Alzheimer's patients based on multimodal perception and a two-layer AI architecture. Background Technology

[0002] Alzheimer's disease (AD) is a progressive neurodegenerative disease. Patients primarily experience memory loss, language impairment, disorientation, and executive function impairment. As the disease progresses, patients gradually lose their ability to live independently and require continuous care support. With the aging population, the number of AD patients continues to rise. Statistics show that the number of Alzheimer's patients in my country has exceeded ten million, placing a heavy burden of care on families and society.

[0003] Currently, intelligent assistance technologies for Alzheimer's patients mainly fall into two categories. One is voice-interaction-based intelligent assistants, where patients obtain information or reminders by asking questions. However, general-purpose voice assistants are not optimized for the cognitive characteristics of Alzheimer's patients. When patients' expressions are unclear or their questions are incomplete, the system struggles to understand their true intentions, and repeated questioning exacerbates the patient's frustration and anxiety. The other category is dedicated cognitive assistance systems, typically employing an end-to-end cloud-based large-scale model architecture, uploading all interaction data to the cloud for processing. This approach suffers from significant real-time bottlenecks; network latency leads to choppy interactions, and complete reliance on cloud services poses a single point of failure risk—the system becomes unavailable if the network is interrupted.

[0004] The main shortcomings of the existing technology are: First, the local terminal lacks independent service execution capabilities and cannot respond quickly to common problems; second, the cloud model lacks a verification and error correction mechanism for the first execution result, and once the model outputs an error, there is a lack of effective automatic correction methods; third, the system is not specifically designed for the special needs of Alzheimer's patients (such as memory decline, exposure to dangerous items, and fall detection), and is more general than specific.

[0005] Therefore, how to construct an intelligent assistance method and system that can guarantee real-time response, achieve intelligent error correction, and is specifically adapted to the cognitive characteristics and safety needs of Alzheimer's patients is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] To address the aforementioned issues, this invention proposes an intelligent assistance system and method for Alzheimer's patients based on multimodal perception and a two-layer AI architecture. This invention achieves a synergistic unity of low-latency response and high-reliability intelligent assistance through edge computing nodes executing restricted service instructions, cloud workflows for result verification and error correction, and a compensation execution mechanism. Simultaneously, it integrates disease-specific adaptation functions such as real-time safety monitoring, fall detection, and memory reminders, providing Alzheimer's patients with 24 / 7, low-burden, and highly secure intelligent assistance services. This addresses the technical problems in existing technologies, such as the lack of independent service execution capabilities on local terminals, the lack of automatic error correction mechanisms in cloud models, and the lack of specific design for the cognitive characteristics of Alzheimer's patients.

[0007] To achieve the objective of this invention, the following technical solution is adopted:

[0008] On the one hand, there is an intelligent assistance method for Alzheimer's patients based on multimodal perception and a two-layer AI architecture, which includes the following steps:

[0009] S1, the wearable terminal collects multimodal input data of the user's current interaction round and sends it to the edge computing server;

[0010] S2, the edge computing server, acting as the first computing node, converts at least one modality of the multimodal input data into a representation that can be processed by a large model, and inputs it into a lightweight large language model deployed locally. The lightweight large language model does not generate natural language responses, but only outputs service call instructions in the form of a structured JSON array within a preset limited service set. The first computing node executes the service call instructions to obtain the first service execution result.

[0011] S3 sends the user's original question, a set of machine-readable service instructions, the first service execution result, and the historical context to the cloud workflow service as a second computing node;

[0012] S4, the cloud workflow service determines whether the result of the initial service execution is sufficient to answer the user's question:

[0013] If the response is sufficient, a final reply will be generated and the correct tag will be attached.

[0014] If the response is insufficient, a hybrid protocol response is output, consisting of natural language intermediate text and machine-readable correction payload. The format of the hybrid protocol response includes at least one machine-readable correction tag pair, which is used to define the start and end positions of the correction payload.

[0015] S5, after receiving the hybrid protocol response, the first computing node does not deserialize the entire text. Instead, it first locates the starting point of the correction load based on the preset control state marker, which includes [Correction: tag]. After locating the starting position of [Correction: tag], it starts from the first curly brace { or square bracket [ after the tag and uses a bracket balancing algorithm based on stack depth balancing to extract the complete and correctly formatted JSON text. The stack depth balancing algorithm counts the depth of curly braces and square brackets respectively. When the depth of the starting symbol returns to zero, the ending position of the JSON text is determined. The extracted complete JSON text is then sent to the deserialization module for parsing to obtain a machine-readable correction load.

[0016] S6, the first computing node reuses at least one multimodal input data segment frozen and retained in the current interaction round, including but not limited to video images, audio segments or IMU data, dynamically assembles the input data segment into the parameter format required by the corresponding service according to the service category in the correction instruction, and then performs compensation execution;

[0017] S7. Determine whether the number of consecutive compensation execution rounds has reached the preset limit. If the preset limit has been reached, output a termination response directly and stop recursively correcting. If the preset limit has not been reached, submit the compensation execution result to the cloud workflow service again to generate a final response, which is then broadcast via voice through the wearable terminal to complete the interactive loop.

[0018] Furthermore, the edge computing server generates a restricted set of machine-readable service instructions, specifically including:

[0019] The edge computing server only outputs service call instructions within a preset set of restricted services. The set of restricted services includes face recognition service, visual recognition service, weather query service, network search service, memory service, and alarm service.

[0020] The edge computing server normalizes the service call instructions according to a fixed priority order, which is: alarm service takes priority over memory service, memory service takes priority over weather query service, weather query service takes priority over network search service, network search service takes priority over face recognition service, and face recognition service takes priority over visual recognition service.

[0021] When the user's intent cannot be recognized, output the null instruction flag [{"none": true}].

[0022] Furthermore, the reuse of the multimodal input data fragment frozen in the current interaction round specifically includes:

[0023] After the current interaction round is formed, the current edge computing server freezes and retains the corresponding environmental awareness data fragment, which includes at least the latest video image.

[0024] During the initial execution or compensation execution phase, the environmental awareness data fragments are dynamically assembled into different forms of input parameters according to different service categories:

[0025] For face recognition services, the video images are assembled into a raw binary image byte stream;

[0026] For visual recognition services, the video images are assembled into Base64 encoded image strings and combined with user questions for input;

[0027] The freezing and reuse mechanism ensures that the same round of question-and-answer and error correction processes are based on the same scenario evidence, reducing the risk of scenario drift caused by re-collection.

[0028] Furthermore, the voice broadcasting step described in S7 includes an asynchronous broadcasting merging and redundancy elimination mechanism based on state association identifiers:

[0029] The system triggers an intermediate state broadcast during the initial service execution waiting phase or error correction phase, and associates the intermediate state broadcast with the final broadcast under the same state association identifier.

[0030] The intermediate broadcast corresponds to the first segment identifier, and the final broadcast corresponds to the second segment identifier;

[0031] When outputting the final broadcast, the system automatically identifies and eliminates prefixes that are repeated with the already broadcast intermediate states, in order to reduce anxiety while waiting for error correction and avoid repeated broadcasts.

[0032] Furthermore, it also includes real-time security monitoring steps:

[0033] Wearable devices continuously capture video streams from a first-person perspective and transmit them to edge computing servers via an independent channel;

[0034] The edge computing server operates independently of the voice interaction process, continuously running an object detection model in an independent thread to detect dangerous items in video frames, and simultaneously running a hand key point recognition model to detect hand position and posture.

[0035] Real-time detection of whether there is spatial overlap between key hand points and the hazardous material detection box; when contact or imminent contact is detected, a high-priority alarm is immediately generated.

[0036] The alarm is broadcast via TTS, employing a high-priority queue-jumping strategy to preempt medium- and low-priority tasks that are currently playing.

[0037] The wearable terminal uses a PSRAM-based frame buffering mechanism during video acquisition and employs a dual-frame buffering and latest frame capture strategy to cache video frames; the edge computing server receives the video frames uploaded by the wearable terminal and performs subsequent recognition processing; the target detection model is a YOLO series target detection model, and the dangerous items include one or more of the following: knives, scissors, medicine bottles, and sources of ignition.

[0038] Furthermore, it also includes fall detection and low-power management steps based on inertial measurement unit data:

[0039] Fall detection steps: A state machine is constructed based on inertial measurement unit data, consisting of "impact trigger → confirmation window → static judgment"; when the acceleration exceeds the first threshold, impact detection is triggered, and the system enters the confirmation window; within the confirmation window, the angular velocity and acceleration are checked to see if the static condition is met; after the static condition is met for a set duration, a fall is confirmed and an alarm is reported; the first threshold is 5.0g, where g represents gravitational acceleration, the confirmation window duration is 2500 milliseconds, and the set duration for the static condition to be met is 1200 milliseconds;

[0040] Low power management steps: When the device remains stationary for a continuous period of time, the audio and video stream upload is automatically paused and the device enters low power mode; during TTS playback and within the protection window after TTS playback ends, entering low power mode is prohibited; the first duration of the continuous stationary period is 10 seconds, and the duration of the protection window after TTS playback ends is 1.2 seconds.

[0041] Furthermore, it also includes the linkage between memory and reminder steps:

[0042] The local intent router identifies schedule-type memory intents in user commands and outputs structured commands containing memory service calls and schedule tags;

[0043] The memory service writes the schedule details into a memory bank and marks them as schedule types;

[0044] The cloud-based workflow identifies the memory as a schedule and extracts the time information and event content. If the time information is complete, a planned reminder event is generated directly. If the time information is ambiguous, a reasonable duration is recommended based on the scenario common sense base. If the time information is obviously unreasonable, a reminder to be confirmed is generated.

[0045] The scheduler is reminded to periodically poll and execute high-priority TTS broadcasts when the system time reaches the trigger time.

[0046] The reminder event and its associated memory entries will be automatically deleted after the broadcast is completed.

[0047] On the other hand, there is an intelligent assistance system for Alzheimer's patients based on multimodal perception and a two-layer AI architecture, including:

[0048] Wearable terminal: integrates at least a camera, microphone, inertial measurement unit and positioning module, used to continuously collect the user's audio and video data and motion data from a first-person perspective;

[0049] Edge computing server: Deployed in the local area network or near-end network where the wearable terminal is located, it serves as the first computing node and maintains a communication connection with the wearable terminal; The edge computing server is used to receive the audio and video data uploaded by the wearable terminal, perform low-latency speech recognition, local intent routing, service orchestration and text-to-speech task scheduling, and submit relevant information to the cloud workflow when complex understanding, result verification or error correction is required;

[0050] Cloud workflow: As a second computing node, it is used to make final decisions, correct errors, and generate responses based on user issues, local routing results, and service execution results;

[0051] The system forms a closed-loop collaborative structure: wearable terminal data collection, edge computing server routing and execution, cloud workflow adjudication and error correction, and edge computing server compensation execution and broadcasting.

[0052] Furthermore, the edge computing server includes a service orchestration module, which is used to parse the structured service call instructions, use an asynchronous concurrency mechanism to call multiple corresponding services simultaneously, set independent timeout control for each service call, ensure that the failure or timeout of a single service does not affect the successful execution of other services, and summarize all execution results to form a unified service execution report; the initial service execution and the compensation execution are both completed through the same service orchestration module.

[0053] Furthermore, the wearable terminal and the edge computing server adopt a communication mechanism based on session identifiers; the edge computing server regenerates session identifiers each time a new interaction link is established, in order to isolate control messages and old broadcast tasks left over from the old session.

[0054] The beneficial effects of this invention are:

[0055] 1. Synergistic Unity of Low Latency and High Reliability: This invention employs a two-tier AI architecture of "edge computing + cloud workflow." Edge computing servers are deployed on computing nodes within the local area network or near-end network where the wearable terminal resides. They process and respond locally to common service requests (such as facial recognition, visual recognition, and weather inquiries), avoiding the network latency and single-point-of-failure risks associated with complete reliance on the cloud. The cloud workflow, acting as a second computing node, is only invoked when complex understanding, result verification, or error correction is required, achieving a balance between real-time performance and intelligence.

[0056] 2. Automatic Error Correction and Closed-Loop Optimization Mechanism: This invention, through the design of a set of machine-readable service instructions and hybrid protocol responses, enables the cloud to verify the initial execution results of edge nodes and output a response containing corrective load when insufficient to answer user questions. After extracting the corrective instructions, the edge node reuses the multimodal data fragments frozen in the current interaction round for compensatory execution, forming a closed loop of "execution → verification → error correction → compensation". This mechanism significantly reduces the impact of model output errors on user experience and improves the robustness of the system.

[0057] 3. Comprehensive Assistance Capabilities Tailored to Alzheimer's Disease: This invention is specifically designed for the unique needs of Alzheimer's patients: the memory bank service within the limited service set is used to record and retrieve the patient's personal information, item locations, and to-do reminders; the alarm service can be linked to hazardous material contact detection and fall detection to generate high-priority alarms in real time; the memory and reminder linkage steps support the automatic recognition and reminder generation of schedule-based memories. These functions together constitute a comprehensive assistance system for Alzheimer's patients.

[0058] 4. Low Interaction Burden and High User Experience: This invention provides patients with appropriate intermediate feedback during error correction waiting periods through intermediate state broadcast merging and redundancy elimination mechanisms, while avoiding repeated broadcasts. Simultaneously, a session-identifier-based communication mechanism isolates control messages left over from previous sessions, ensuring clarity in multi-turn dialogues. These designs reduce patients' cognitive burden and waiting anxiety, enhancing the naturalness and friendliness of the interaction.

[0059] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0060] Figure 1 This is a flowchart of the two-layer AI error correction closed loop of the present invention;

[0061] Figure 2 This is a sequence diagram of the service orchestration and concurrent execution of this invention;

[0062] Figure 3 This is the overall system architecture diagram of the present invention; Detailed Implementation

[0063] Example 1

[0064] This embodiment is an intelligent assistance method for Alzheimer's patients based on multimodal perception and a two-layer AI architecture, such as... Figures 1 to 2 As shown, it includes the following steps:

[0065] S1, the wearable terminal collects multimodal input data of the user's current interaction round and sends it to the edge computing server.

[0066] S2, the edge computing server, acting as the first computing node, converts at least one modality of the multimodal input data into a representation that can be processed by a large model, and inputs it into a lightweight large language model deployed locally. The lightweight large language model does not generate natural language responses, but only outputs service call instructions in the form of a structured JSON array within a preset limited service set. The first computing node executes the service call instructions to obtain the first service execution result.

[0067] The edge computing server generates a restricted set of machine-readable service instructions, specifically including:

[0068] The edge computing server only outputs service call instructions within a preset set of restricted services, which includes face recognition service, visual recognition service, weather query service, network search service, memory service, and alarm service.

[0069] The edge computing server normalizes the service call instructions according to a fixed priority order, which is as follows: alarm service takes precedence over memory service, memory service takes precedence over weather query service, weather query service takes precedence over network search service, network search service takes precedence over face recognition service, and face recognition service takes precedence over visual recognition service.

[0070] When the user's intent cannot be recognized, output the null instruction flag [{"none": true}].

[0071] S3 sends the user's original question, a set of machine-readable service instructions, the initial service execution result, and historical context to the cloud workflow service as a second computing node.

[0072] S4, the cloud workflow service determines whether the result of the initial service execution is sufficient to answer the user's question:

[0073] If the response is sufficient, a final reply will be generated and the correct tag will be attached.

[0074] If the response is insufficient, a hybrid protocol response is output, consisting of natural language intermediate text and machine-readable correction payload. The format of the hybrid protocol response includes at least one machine-readable correction tag pair, which is used to define the start and end positions of the correction payload.

[0075] S5, after receiving the hybrid protocol response, the first computing node does not deserialize the entire text. Instead, it first locates the starting point of the correction load based on a preset control state marker, which includes [Correction: tag]. After locating the starting position of [Correction: tag], it extracts the complete and correctly formatted JSON text starting from the first curly brace { or square bracket [ after the tag) using a bracket balancing algorithm based on stack depth balancing. The stack depth balancing algorithm counts the depth of curly braces and square brackets respectively. When the depth of the starting symbol returns to zero, it determines the end position of the JSON text. The extracted complete JSON text is then sent to the deserialization module for parsing to obtain a machine-readable correction load.

[0076] S6, the first computing node reuses at least one multimodal input data segment frozen and retained in the current interaction round, including but not limited to video images, audio segments or IMU data, dynamically assembles the input data segment into the parameter format required by the corresponding service according to the service category in the correction instruction, and then performs compensation execution.

[0077] S7. Determine whether the number of consecutive compensation execution rounds has reached the preset limit. If the preset limit has been reached, output a termination response directly and stop recursively correcting. If the preset limit has not been reached, submit the compensation execution result to the cloud workflow service again to generate a final response, which is then broadcast via voice through the wearable terminal to complete the interactive loop.

[0078] The voice broadcasting steps include an asynchronous broadcasting merging and redundancy elimination mechanism based on state association identifiers: the system triggers intermediate state broadcasting during the initial service execution waiting phase or error correction phase, and associates the intermediate state broadcasting with the final broadcasting under the same state association identifier; the intermediate state broadcasting corresponds to the first segment identifier, and the final broadcasting corresponds to the second segment identifier; when outputting the final broadcasting, the system automatically identifies and eliminates prefix content that is repeated with the already broadcast intermediate state, so as to reduce error correction waiting anxiety and avoid repeated broadcasting.

[0079] The system of this invention also includes a fault-tolerant processing link for edge cases. When a cloud workflow call times out or an anomaly occurs, the edge side stops the current automatic error correction process and outputs a preset anomaly prompt to the front end or interaction link, without continuing to recursively initiate corrections. When the number of consecutive correction rounds reaches a preset limit, the system outputs a preset failover text and terminates further corrections to prevent the error correction link from extending indefinitely. When the broadcast link does not receive a confirmation message or playback completion message within a preset time, the server side performs timeout cleanup on the corresponding suspended broadcast task and releases the waiting state. Thus, the system not only protects the main links of normal question and answer and automatic error correction, but also provides a convergent processing mechanism for edge scenarios such as cloud timeouts, continuous correction failures, and abnormal broadcast timing.

[0080] Example 2

[0081] Based on Embodiment 1, a preferred embodiment of the present invention further includes a real-time security monitoring step:

[0082] Wearable devices continuously capture video streams from a first-person perspective and transmit them to edge computing servers via an independent channel.

[0083] The edge computing server operates independently of the voice interaction process, continuously running an object detection model in an independent thread to detect dangerous items in video frames, and simultaneously running a hand key point recognition model to detect hand position and posture.

[0084] It can determine in real time whether there is spatial overlap between key points of the hand and the hazardous material detection box. When it is determined that there is contact or imminent contact, a high-priority alarm is generated immediately.

[0085] The alarm is broadcast via TTS, employing a high-priority queue-jumping strategy to preempt medium- and low-priority tasks that are currently playing.

[0086] The wearable terminal uses a PSRAM-based frame buffering mechanism during video acquisition and employs a dual-frame buffering and latest frame capture strategy to cache video frames; the edge computing server receives the video frames uploaded by the wearable terminal and performs subsequent recognition processing; the target detection model is a YOLO series target detection model, and the dangerous items include one or more of the following: knives, scissors, medicine bottles, and sources of ignition.

[0087] Example 3

[0088] Based on the above embodiments, a preferred embodiment of the present invention further includes a fall detection and low-power management step based on inertial measurement unit data:

[0089] Fall detection steps: Construct a state machine based on inertial measurement unit data, consisting of "impact trigger → confirmation window → static judgment"; when the acceleration exceeds the first threshold, impact detection is triggered, and the system enters the confirmation window; within the confirmation window, it checks whether the angular velocity and acceleration meet the static condition; after the static condition is met for a set duration, a fall is confirmed and an alarm is reported; the first threshold is 5.0g, where g represents gravitational acceleration, the confirmation window duration is 2500 milliseconds, and the set duration for the static condition to be met is 1200 milliseconds.

[0090] Low power management steps: When the device remains stationary for a continuous period of time, the audio and video stream upload is automatically paused and the device enters low power mode; during TTS playback and within the protection window after TTS playback ends, entering low power mode is prohibited; the first duration of the continuous stationary period is 10 seconds, and the duration of the protection window after TTS playback ends is 1.2 seconds.

[0091] Example 4

[0092] Based on the above embodiments, a preferred embodiment of the present invention further includes the following memory and reminder linkage steps:

[0093] The local intent router recognizes the schedule-type memory intent in the user's instructions and outputs a structured instruction containing memory service calls and schedule tags.

[0094] The memory service writes the schedule content into a memory bank and marks it as a schedule type.

[0095] The cloud-based workflow identifies the memory as a schedule and extracts the time information and event content. If the time information is complete, a planned reminder event is generated directly. If the time information is ambiguous, a reasonable duration is recommended based on the scenario common sense base. If the time information is obviously unreasonable, a reminder to be confirmed is generated.

[0096] The scheduler is reminded to periodically poll and execute high-priority TTS broadcasts when the system time reaches the trigger time.

[0097] The reminder event and its associated memory entries will be automatically deleted after the broadcast is completed.

[0098] Example 5

[0099] Based on the above embodiments, this embodiment is a preferred embodiment of a two-layer AI automatic error correction closed loop, such as... Figure 1 As shown.

[0100] The following example, using a patient asking "What is this?", illustrates the automatic error correction process of this invention when the local routing results are inaccurate.

[0101] S201, the patient points to an object on the table and asks, "What is this?" After the edge computing server completes speech recognition, it submits the recognized text to the local intent routing module. Because the question is short and ambiguous, the local routing module may output an inaccurate result in some cases, for example:

[0102] [{"service":"face"}]

[0103] S202, the service orchestrator first executes the face service according to the initial route described above. If there is no recognizable face in the current screen, the face service returns a failure result or a result indicating that it cannot answer the user's question.

[0104] S203, the edge computing server sends the user's question, local initial routing result, first execution result, and recent context to the cloud workflow. Under preset prompt word rules, the cloud workflow checks whether the current result is sufficient to answer the question, based on the user's question, local initial routing result, first execution result, and recent context. If it determines that the current result cannot answer the real question, it returns a response with an error execution label and a correction command label, for example:

[0105] Let me take another look. [Execute: error][Correction: [{"service":"vision"}]]

[0106] S204. After receiving the above response, the edge computing server does not directly perform JSON deserialization on the entire response text. The reason is that cloud workflow responses typically contain both natural language prefixes and protocol tags, such as "Let me take another look. [Execute: error][Correction: ...]". Directly deserializing the entire text would fail due to the presence of natural language prefixes, while using simple regular expression matching or truncation by the first ] could lead to incomplete extraction if nested objects or arrays exist within the correction command. Therefore, the system first locates the beginning of the [Correction:] tag, then extracts the complete JSON text starting from the first { or [ after the tag, using a stack depth balancing method for curly braces and square brackets. After extraction, it consumes the closing bracket of the outer tag before sending the complete JSON text to the deserialization module, thus ensuring stable parsing of the correction command.

[0107] S205, before entering the correction execution, the system generates a unified group_id for this round of error correction and adds intermediate state texts such as "Let me take a closer look" to the broadcast link according to segment=intermediate. If no intermediate state broadcast has occurred in the local routing stage, then the intermediate state broadcast will be performed at this time; if the intermediate state has already been broadcast in the local routing stage, then it will not be broadcast again in this round. The server side writes the group_id into the intermediate state cache for use in subsequent final state broadcasts to eliminate duplicate prefixes.

[0108] S206, after parsing the correction command, the system calls the service orchestrator again to execute it. In this embodiment, the correction command is a vision service, so the system reuses the video frames retained in this round, re-injects image_source and user_question, and then performs visual recognition. If the correction command involves the face service, image_bytes are automatically padded. If the correction command involves the search service, the system prioritizes reusing the current user question and can combine explicit entities in the recent context or recent search topics to padded the query. If the correction command involves the alert service, the system automatically fills in user_text and the source field. The initial execution and the correction execution are both completed through the same service orchestrator, thereby maintaining consistency in scheduling rules.

[0109] S207 After correcting the service return result, the system resubmits to the cloud workflow, which outputs the final response with the [Execute: correct] tag. If the cloud continuously returns correction commands up to the preset limit, the system will no longer continue recursively correcting, but will output a preset failover text to prevent the error correction chain from extending indefinitely.

[0110] S208, the edge computing server sends the final response to the broadcasting link in the following manner: If an intermediate state has been broadcast before in this round, the final broadcast uses the aforementioned group_id and is marked as segment=final; when generating the final broadcast text, the system checks whether the intermediate state text has been broadcast with the same prefix. If there is a duplicate, only the newly added information is broadcast. If no intermediate state has been broadcast before in this round, the final response for this round is output in a single-segment broadcast manner.

[0111] S209, thus forming a two-layer AI automatic error correction closed loop of "local structured routing, initial service execution, cloud error judgment, structured correction command, automatic parameter supplementation and re-execution, and final broadcast". Compared with the method of restarting the entire round of free question and answer after discovering that the first round of response is inaccurate, this implementation reuses the video frames, user questions and existing service results retained in the current round during the error correction stage, and only supplements parameters and re-executes the structured correction command returned by the cloud, thereby avoiding repeated image capture, repeated assembly of service requests and unnecessary link back-off; especially when the patient's question is short, repetitive or has expression deviations, it can complete the correction with less additional overhead, improve the controllability of error correction latency and the stability of edge-side execution.

[0112] Optionally, for search correction scenarios, the system can also complete the query based on explicit entities in the recent context or recent search topics to reduce the probability of misjudging the query again. Optionally, for alert correction scenarios, the system can backfill the current user's original text into user_text to ensure that the corrected alert command does not deviate from the original semantics.

[0113] Example 6

[0114] This embodiment describes a system for intelligent assistance methods for Alzheimer's patients based on multimodal perception and a two-layer AI architecture, such as... Figure 3 As shown, the system includes:

[0115] Wearable terminal: integrates at least a camera, microphone, inertial measurement unit and positioning module, used to continuously collect the user's audio and video data and motion data from a first-person perspective.

[0116] Edge computing server: Deployed in the local area network or near-end network where the wearable terminal is located, it serves as the first computing node and maintains a communication connection with the wearable terminal; the edge computing server is used to receive the audio and video data uploaded by the wearable terminal, perform low-latency speech recognition, local intent routing, service orchestration and text-to-speech task scheduling, and submit relevant information to the cloud workflow when complex understanding, result verification or error correction is required.

[0117] The edge computing server includes a service orchestration module, which parses the structured service call instructions, uses an asynchronous concurrency mechanism to call multiple corresponding services simultaneously, sets independent timeout control for each service call, and ensures that the failure or timeout of a single service does not affect the successful execution of other services, and summarizes all execution results to form a unified service execution report; the initial service execution and the compensation execution are both completed through the same service orchestration module.

[0118] Cloud Workflow: As a second computing node, it is used to make final decisions, correct errors, and generate responses based on user issues, local routing results, and service execution results.

[0119] The system forms a closed-loop collaborative structure: wearable terminal data collection, edge computing server routing and execution, cloud workflow adjudication and error correction, and edge computing server compensation execution and broadcasting.

[0120] The wearable terminal and the edge computing server use a communication mechanism based on session identifiers; the edge computing server regenerates session identifiers each time a new interaction link is established, in order to isolate control messages and old broadcast tasks left over from the old session.

[0121] In a preferred embodiment of the present invention, after the terminal starts up, it first accesses the wireless network, and then listens for UDP broadcasts on the local area network to discover the address of the edge computing server. The server discovery message may contain a preset identifier HereIsServer. After discovering the server, the terminal establishes two WebSocket channels, / ws / audio and / ws / camera, respectively. For the audio link, the terminal first sends an AUDIO_READY handshake message to the server, and the server returns OK:AUDIO_READY:<session_id> The terminal then sends the START command along with the session_id, and the server returns OK:STARTED:<session_id> This establishes the current audio session. The server regenerates the session_id each time a new audio link is established to isolate control messages and old playback tasks left over from the old session.

[0122] The camera captures images at a VGA resolution of 640×480, encoded in JPEG format, with the frame capture mode set to CAMERA_GRAB_LATEST. Microphone data is captured via PDM and converted to 16kHz, 16-bit, mono PCM, then divided into 640-byte audio blocks at 20ms intervals. Both the terminal and server sides employ a queue-based decoupling structure. When a queue is full, the oldest data is discarded first to ensure the system continuously processes the latest scene and audio.

[0123] After receiving the speech-recognized text, the edge computing server does not directly generate a natural language response from the local model. Instead, it outputs structured service call commands solely through the local intent routing module. The current restricted service set includes vision, face, weather, search, memory, and alert. Before execution, these services are normalized in a fixed priority order: alert > memory > weather > search > face > vision. The edge computing server then automatically completes the parameters required for each service and hands them over to a unified service orchestrator for execution. The service orchestrator itself is kept as stateless as possible, with long-term state only maintained in the service registry. Intermediate variables for each execution batch reside in a local scope so that they can be reused concurrently by multiple requests.

[0124] The local intent routing module consists of an edge-side routing processing module and a command set constraint file. At its core, it calls the local llama-server service via an OpenAI-compatible HTTP interface, loading the routing model file `router-qwen35-Q4_K_M.gguf`, which has been quantized using LoRA tuning. Edge nodes input fixed system prompts into the model, which constrain at least the following: only valid JSON arrays of six service types (vision, face, weather, search, memory, and alert) are allowed; outputting natural language explanations, thought processes, Markdown, or empty arrays is prohibited; and the output order of multiple commands is specified. After model inference, the system also performs structure verification and normalization on the output results. When the model output is empty, not an array, contains illegal services, or is merely unauthorized text, it is normalized to `[{"none":true}]`, thus ensuring that the commands entering the service execution chain are always restricted structured commands, not freely generated text.

[0125] Considering the actual assistance needs of Alzheimer's patients, the aforementioned set of limited services is not arbitrarily set for general chat scenarios. Specifically, the memory service records and retrieves the patient's personal information, object locations, relationships, and to-do reminders, adapting to scenarios of memory decline and repetitive verification; the alert service handles high-risk events such as requests for help, obvious discomfort, and falls, and can further link to the caregiver notification chain; the terminal-side IMU and positioning module can provide fall status and location information, respectively. Therefore, the local routing, error correction, and broadcasting links in this invention are not general question-and-answer processes detached from the disease, but rather constructed around the memory assistance, safe care, and low-burden interaction needs of Alzheimer's patients.

[0126] As a system implementation, the wearable terminal side includes at least an audio / video acquisition unit, a wireless communication unit, and a voice broadcasting unit; the edge computing server side includes at least an audio receiving module, a video receiving module, a local intent routing module, an environmental awareness data caching module, a service orchestration module, and a text-to-speech management module; and the cloud side includes at least a result verification and correction module. During this process, the recognized text output by the audio receiving module enters the local intent routing module, the current-round image data output by the video receiving module enters the environmental awareness data caching module, the machine-readable service instructions output by the local intent routing module enter the service orchestration module, the initial execution result output by the service orchestration module enters the cloud-side result verification and correction module, and the final response or correction instruction returned by the cloud-side result verification and correction module is then transmitted back to the edge-side text-to-speech management module, thus forming a closed-loop collaborative structure between the terminal, edge, and cloud.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features, and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent assistance to Alzheimer's patients based on multimodal perception and a two-layer AI architecture, characterized in that, Includes the following steps: S1, the wearable terminal collects multimodal input data of the user's current interaction round and sends it to the edge computing server; S2, the edge computing server, acting as the first computing node, converts at least one modality of the multimodal input data into a representation that can be processed by a large model, and inputs it into a lightweight large language model deployed locally. The lightweight large language model does not generate natural language responses, but only outputs service call instructions in the form of a structured JSON array within a preset limited service set. The first computing node executes the service call instructions to obtain the first service execution result. S3 sends the user's original question, a set of machine-readable service instructions, the first service execution result, and the historical context to the cloud workflow service as a second computing node; S4, the cloud workflow service determines whether the result of the initial service execution is sufficient to answer the user's question: If the response is sufficient, a final reply will be generated and the correct tag will be attached. If the response is insufficient, a hybrid protocol response is output, consisting of natural language intermediate text and machine-readable correction payload. The format of the hybrid protocol response includes at least one machine-readable correction tag pair, which is used to define the start and end positions of the correction payload. S5, after receiving the hybrid protocol response, the first computing node does not deserialize the entire text. Instead, it first locates the starting point of the correction load based on the preset control state marker, which includes [Correction: tag]. After locating the starting position of [Correction: tag], it starts from the first curly brace { or square bracket [ after the tag and uses a bracket balancing algorithm based on stack depth balancing to extract the complete and correctly formatted JSON text. The stack depth balancing algorithm counts the depth of curly braces and square brackets respectively. When the depth of the starting symbol returns to zero, the ending position of the JSON text is determined. The extracted complete JSON text is then sent to the deserialization module for parsing to obtain a machine-readable correction load. S6, the first computing node reuses at least one multimodal input data segment frozen and retained in the current interaction round, including but not limited to video images, audio segments or IMU data, dynamically assembles the input data segment into the parameter format required by the corresponding service according to the service category in the correction instruction, and then performs compensation execution; S7, determine whether the number of consecutive compensation execution rounds has reached the preset limit. If the preset limit has been reached, output the termination response directly and stop recursively correcting. If the preset upper limit is not reached, the compensation execution result will be submitted to the cloud workflow service again to generate a final response, which will be broadcast via voice through the wearable terminal to complete the interactive loop. The S7 also includes fault tolerance processing for edge cases: when a cloud workflow call times out or an exception occurs, the edge computing server stops the current automatic error correction process and outputs a preset exception prompt, without continuing to recursively initiate correction; when the broadcast link does not receive a confirmation message or a playback completion message within a preset time, the edge computing server performs timeout cleanup on the corresponding suspended broadcast task and releases the waiting state.

2. The intelligent assistance method for Alzheimer's patients based on multimodal perception and a two-layer AI architecture according to claim 1, characterized in that, The edge computing server generates a restricted set of machine-readable service instructions, specifically including: The edge computing server only outputs service call instructions within a preset set of restricted services. The set of restricted services includes face recognition service, visual recognition service, weather query service, network search service, memory service, and alarm service. The edge computing server normalizes the service call instructions according to a fixed priority order, which is: alarm service takes priority over memory service, memory service takes priority over weather query service, weather query service takes priority over network search service, network search service takes priority over face recognition service, and face recognition service takes priority over visual recognition service. When the user's intent cannot be recognized, output the null instruction flag [{"none": true}].

3. The intelligent assistance method for Alzheimer's patients based on multimodal perception and a two-layer AI architecture according to claim 1, characterized in that, The reuse of the multimodal input data fragment frozen in the current interaction round specifically includes: After the current interaction round is formed, the current edge computing server freezes and retains the corresponding environmental awareness data fragment, which includes at least the latest video image. During the initial execution or compensation execution phase, the environmental awareness data fragments are dynamically assembled into different forms of input parameters according to different service categories: For face recognition services, the video images are assembled into a raw binary image byte stream; For visual recognition services, the video images are assembled into Base64 encoded image strings and combined with user questions for input; The freezing and reuse mechanism ensures that the same round of question-and-answer and error correction processes are based on the same scenario evidence, reducing the risk of scenario drift caused by re-collection.

4. The intelligent assistance method for Alzheimer's patients based on multimodal perception and a two-layer AI architecture according to claim 1, characterized in that, The voice broadcasting step described in S7 includes an asynchronous broadcasting merging and redundancy elimination mechanism based on state association identifiers: The system triggers an intermediate state broadcast during the initial service execution waiting phase or error correction phase, and associates the intermediate state broadcast with the final broadcast under the same state association identifier. The intermediate broadcast corresponds to the first segment identifier, and the final broadcast corresponds to the second segment identifier; When outputting the final broadcast, the system automatically identifies and eliminates prefixes that are repeated with the already broadcast intermediate states, in order to reduce anxiety while waiting for error correction and avoid repeated broadcasts.

5. The intelligent assistance method for Alzheimer's patients based on multimodal perception and a two-layer AI architecture according to claim 1, characterized in that, It also includes real-time security monitoring steps: Wearable devices continuously capture video streams from a first-person perspective and transmit them to edge computing servers via an independent channel; The edge computing server operates independently of the voice interaction process, continuously running an object detection model in an independent thread to detect dangerous items in video frames, and simultaneously running a hand key point recognition model to detect hand position and posture. Real-time detection of whether there is spatial overlap between key hand points and the hazardous material detection box; when contact or imminent contact is detected, a high-priority alarm is immediately generated. The alarm is broadcast via TTS, employing a high-priority queue-jumping strategy to preempt medium- and low-priority tasks that are currently playing. The wearable terminal uses a PSRAM-based frame buffering mechanism during video acquisition, and employs a double-frame buffering and latest frame capture strategy to cache video frames. The edge computing server receives video frames uploaded by the wearable terminal and performs subsequent recognition processing. The target detection model is the YOLO series target detection model. The dangerous items include one or more of the following: knives, scissors, medicine bottles, and sources of ignition.

6. The intelligent assistance method for Alzheimer's patients based on multimodal perception and a two-layer AI architecture according to claim 5, characterized in that, It also includes fall detection and low-power management steps based on inertial measurement unit data: Fall detection steps: Construct a state machine based on inertial measurement unit data, consisting of "impact trigger → confirmation window → static judgment"; when the acceleration exceeds the first threshold, impact detection is triggered, and the system enters the confirmation window; within the confirmation window, it checks whether the angular velocity and acceleration meet the static condition; after the static condition is met for a set duration, a fall is confirmed and an alarm is reported; the first threshold is 5.0g, where g represents gravitational acceleration, the confirmation window duration is 2500 milliseconds, and the set duration for the static condition to be met is 1200 milliseconds; Low power management steps: When the device remains stationary for a continuous period of time, the audio and video stream upload is automatically paused and the device enters low power mode; during TTS playback and within the protection window after TTS playback ends, entering low power mode is prohibited; the first duration of the continuous stationary period is 10 seconds, and the duration of the protection window after TTS playback ends is 1.2 seconds.

7. The intelligent assistance method for Alzheimer's patients based on multimodal perception and a two-layer AI architecture according to claim 6, characterized in that, It also includes the linkage between memory and reminder steps: The local intent router identifies schedule-type memory intents in user commands and outputs structured commands containing memory service calls and schedule tags; The memory service writes the schedule details into a memory bank and marks them as schedule types; The cloud-based workflow identifies the memory as a schedule type and extracts the time information and event content. If the time information is complete, a scheduled reminder event will be generated directly. If the time information is ambiguous, a reasonable duration will be recommended based on the scenario common sense database; If the time information is clearly unreasonable, generate a confirmation reminder. The scheduler is reminded to periodically poll and execute high-priority TTS broadcasts when the system time reaches the trigger time. The reminder event and its associated memory entries will be automatically deleted after the broadcast is completed.

8. An intelligent assistance system for Alzheimer's patients based on multimodal perception and a two-layer AI architecture, wherein the method described in any one of claims 1-7 is characterized in that, include: Wearable terminal: integrates at least a camera, microphone, inertial measurement unit and positioning module, used to continuously collect the user's audio and video data and motion data from a first-person perspective; Edge computing server: Deployed in the local area network or near-end network where the wearable terminal is located, it serves as the first computing node and maintains a communication connection with the wearable terminal; The edge computing server is used to receive the audio and video data uploaded by the wearable terminal, perform low-latency speech recognition, local intent routing, service orchestration and text-to-speech task scheduling, and submit relevant information to the cloud workflow when complex understanding, result verification or error correction is required; Cloud workflow: As a second computing node, it is used to make final decisions, correct errors, and generate responses based on user issues, local routing results, and service execution results; The system forms a closed-loop collaborative structure: wearable terminal data collection, edge computing server routing and execution, cloud workflow adjudication and error correction, and edge computing server compensation execution and broadcasting.

9. The intelligent assistance system for Alzheimer's patients based on multimodal perception and a two-layer AI architecture according to claim 8, characterized in that, The edge computing server includes a service orchestration module, which parses the structured service call instructions, uses an asynchronous concurrency mechanism to call multiple corresponding services simultaneously, sets independent timeout control for each service call, and ensures that the failure or timeout of a single service does not affect the successful execution of other services, and summarizes all execution results to form a unified service execution report; the initial service execution and the compensation execution are both completed through the same service orchestration module.

10. The intelligent assistance system for Alzheimer's patients based on multimodal perception and a two-layer AI architecture according to claim 8, characterized in that, The wearable terminal and the edge computing server use a communication mechanism based on session identifiers; the edge computing server regenerates session identifiers each time a new interaction link is established, in order to isolate control messages and old broadcast tasks left over from the old session.

Images